homepage

Many C functions have bytes argument (char* type) but the encoding is not documented. If would not be a problem if the encoding was always the same, but it is not. Examples:
 - format of PyUnicode_FromFormat() should be encoded as ISO-8859-1
 - filename of PyParser_ASTFromString() should be encoded as utf-8
 - filename of PyErr_SetFromErrnoWithFilename() should be encoded to the filesystem encoding (with strict error handler, and not surrogateescape)
 - 's' argument of PyParser_ASTFromString() should be encoded as utf-8 if PyPARSE_IGNORE_COOKIE flag is set, otherwise the parser checks for #coding:xxx cookie (if there is no cookie, utf-8 is used)
Attached patch is a try to document most low level functions. I choosed to add the name of function arguments in the headers because I consider that a header can be used as a quick documentation. I only touched .c files to change argument names.
It is hard to get the right encoding, so I cannot ensure that my patch is correct. My patch is just a draft.
I don't know if "encoded to utf-8" is the right expression. Or should it be "decoded as utf-8"?

I think either of these is correct:
- a UTF-8-encoded string
- a string encoded in UTF-8

> I think either of these is correct:
> - a UTF-8-encoded string
> - a string encoded in UTF-8
Possibly use the word "buffer" here, rather than "string", as "string" may suggest the "str" type.
Or even: "NUL-terminated buffer of UTF-8-encoded bytes", or whatnot.
(sorry for bikeshedding)

Better specifying requirements is good. A few comments:
- The second argument is an error message; it is converted to a string object.
+ The second argument is an error message; it is decoded to a string object
+ with ``'utf-8'`` encoding.
 
I would write the change as
+ The second argument is a utf-8 encoded error message; it is decoded to a string object. 
I the second part (what the function will do with the arg) really needed? I think in the current version, it serves to indirectly specify that the arg in not to be a string, but bytes. If the specific encoding required is specified, that also says bytes, making 'will be decoded' redundant and irrelevant.
-------------------------------
+ a Python exception (class, not an instance). *format* should be a string
+ encoded to ISO-8859-1, containing format codes, 
*format* should be ISO-8859-1 encoded bytes containing format codes,
although I am not clear about the implications of that. Are not all format code ascii chars?
--------------------------------
I do not really like 'encoded to', but 'decoded to' is wrong. 'will be decoded from xxx bytes' is better. I think there should be a general discussion somewhere about bytes arguments and the terminology that will be used.

About PyErr_Format() and PyUnicode_FromFormat*() encoding: it's not exactly ISO-8859-1... there is a bug => issue #9769.

#6543 changed the encoding of the filename argument of PyRun_SimpleFileExFlags() (and all functions based on PyRun_SimpleFileExFlags) and c_filename attribute of the compiler (private) structure in Python 3.1.3: use utf-8 in strict mode instead of filesystem encoding with surrogateescape.

A (probably crazy) idea that just occurred to me:
 typedef char utf8_bytes;
 typedef char iso8859_1_bytes;
 typedef char fsenc_bytes;
then specify the encoding in the type signature of the API e.g.:
- int PyRun_SimpleFile(FILE *fp, const char *filename)
+ int PyRun_SimpleFile(FILE *fp, const fsenc_bytes *filename)

> A (probably crazy) idea that just occurred to me:
> typedef char utf8_bytes;
> typedef char iso8859_1_bytes;
> typedef char fsenc_bytes;
I like it! Let's see how far we can get without iso8859_1_bytes, though. (It is likely to be locale_bytes anyways.) There are a few places where we'll need ascii_bytes.
The added benefit is that we can make these typedefs unsigned char and avoid char signness being ambiguous. We will also need to give the typedefs the Py_ prefix.
And an obligatory bikesheding comment: if we typedef char, we should use singular form. Or we can typedef char* Py_utf8_bytes.

r87504 documents encodings of error functions.
r87505 documents encodings of unicode functions.
r87506 documents encodings of AST, compiler, parser and PyRun functions.

While documenting encodings, I found two issues: #10778 and #10779.

Victor,
Here is an interesting case for your collection: PyDict_GetItemString. Note that it is documented as not setting error, but in fact it may if encoding fails. This rarely an issue because most uses of PyDict_GetItemString are with an ASCII string literal.

> Here is an interesting case for your collection: PyDict_GetItemString.
It's easier to guess the encoding of such function: Python 3 always use UTF-8, but yes, the encoding should be documented.
I documented many functions, directly in the header files, and sometimes also in the reST documentation.
I close this issue because I consider it as done. If you would like to document the encoding of some specific functions, please open new issues.