homepage

A colleague discovered an interesting implementation different between C-defined functions and Python-defined functions called with **kws arguments. He tried to do this:
>>> from collections import defaultdict
>>> '{foo}{bar}'.format(**defaultdict(str))
''
He wondered how this could work, especially given:
>>> def what(**kws):
... print(type(kws), repr(kws))
... 
>>> what(**defaultdict(str))
<class 'dict'> {}
The Language Reference clearly states that when what() is called, a new dictionary is create, which is populated by keyword arguments not consumed by positional args.
http://docs.python.org/3/reference/compound_stmts.html#function-definitions
http://docs.python.org/3/reference/expressions.html#calls
What isn't described is the behavior when the function is defined in C using METH_KEYWORDS. In that case, the object passed through the ** argument is passed unscathed to the underlying methodobject defined to take METH_KEYWORDS. str.format() is of the latter type so it gets the actual defaultdict. what() of course gets the defined behavior.
It would be nice if the implementation details of the function did not leak into this behavior, but this is admittedly a corner case of the CPython implementation. I haven't checked any of the alternative implementations to see what they do. My guess is that CPython won't change for practical and backward compatibility reasons, thus I've opened this issue as a documentation bug. At the very least, an implementation note in the Language Reference should be added.

I'll confirm that PyPy raises a KeyError on the format() code.

Yet one difference is that kwargs can be NULL in C function if no keyword arguments are supplied.
Here is a patch that adds tests for exact types of args and kwargs arguments in C functions.
I think that we should keep passing NULL as kwargs for performance reasons. But shouldn't we convert dict subclass to exact dict? I think there are good reasons to do this. Some functions can save kwargs for latter use. And this makes the implementation more consistent with PyPy.
If defaultdict behavior is needed for formatting, there is format_map().

Proposed patch converts var-keyword arguments to exact dict. Since it changes the behavior of str.format() it can be pushed only in 3.6.

Eric should chime in on the str.format() implications.

I think it's okay to change this as far as str.format() goes.
Interestingly enough, the motivating case to add str.format_map() in issue 6081 was exactly this kind of code. Although I notice that the example in that issue, which it shows as failing, works in both 2.7 and 3.4 (the only versions I have handy). So I'm not sure what changed after 2009 to cause that code to start working, but I'd be okay with it breaking again. But maybe we should track it down, in case it was deliberately changed and is important to someone.
In any event, since str.format_map() is well-known (to me, at least!), and is the approved way of getting the behavior of passing a specific map in to the format machinery, I'm okay with changing **dict to create an exact new dict, at least as far as str.format() goes. I can't speak to the overall implications (such as other APIs and performance).
I agree it would be a 3.6 only change (and I've change the versions to reflect that).

On May 06, 2016, at 03:41 PM, Eric V. Smith wrote:
>I agree it would be a 3.6 only change (and I've change the versions to
>reflect that).
The reason the other versions were included was because this was originally
reported as a documentation bug. It should probably still be documented in
those older releases.

Okay. Adding back 3.5 and 2.7, in case someone wants to document the behavior there. I'm not sure if we fix docs in 3.4 still.

str.format() uses PyDict_GetItem() in 2.7, but PyObject_GetItem() in 3.x. A comment before:
 /* Use PyObject_GetItem instead of PyDict_GetItem because this
 code is no longer just used with kwargs. It might be passed
 a non-dict when called through format_map. */
Thus the behavior of str.format() was changed by accident.

New changeset e4835b1ed7b1 by Serhiy Storchaka in branch 'default':
Issue #18531: Single var-keyword argument of dict subtype was passed
https://hg.python.org/cpython/rev/e4835b1ed7b1 

I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely explains things, because the code in issue 6081 also works in 2.7.
But I don't think it's terribly important in any event, as long as we understand the 3.x difference. Thanks for researching!

> I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely
> explains things, because the code in issue 6081 also works in 2.7.
It doesn't work to me in current develop 2.7.11+. What version did you test?

Now this is just the documentation issue. I think the change of the behavior in 3.6 should be documented too, but I even don't know where.

Oops, sorry. I'm an idiot. I was running in a venv where python was python3. Checking with my installed 2.7.10, I see the expected error.

New changeset e3283b4ae61d by Serhiy Storchaka in branch '3.5':
Backported tests for issue #18531.
https://hg.python.org/cpython/rev/e3283b4ae61d
New changeset 621e0a3249c2 by Serhiy Storchaka in branch '2.7':
Backported tests for issue #18531.
https://hg.python.org/cpython/rev/621e0a3249c2 

This seems fixed in 3.11:
>>> from collections import defaultdict
>>> '{foo}{bar}'.format(**defaultdict(str))
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
KeyError: 'foo'