homepage

On Windows x64, we get the following warning:
..\Objects\codeobject.c(106): warning C4244: '=' : conversion from 'Py_ssize_t' to 'unsigned char', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
Code:
 unsigned char *cell2arg = NULL;
 Py_ssize_t total_args = argcount + kwonlyargcount +
 ((flags & CO_VARARGS) != 0) + ((flags & CO_VARKEYWORDS) != 0);
 PyObject *cell = PyTuple_GET_ITEM(cellvars, i);
 for (j = 0; j < total_args; j++) {
 PyObject *arg = PyTuple_GET_ITEM(varnames, j);
 if (!PyUnicode_Compare(cell, arg)) {
 ====> cell2arg[i] = j; <===== HERE
 used_cell2arg = 1;
 break;
 }
 }
total_args is not checked for being smaller than 256.
Related issue: #9566.

Similar issue:
..\Objects\funcobject.c(636): warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
..\Objects\funcobject.c(637): warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
..\Objects\funcobject.c(637): warning C4244: 'function' : conversion from 'Py_ssize_t' to 'int', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
Extract of function_call() function:
 result = PyEval_EvalCodeEx(
 PyFunction_GET_CODE(func),
 PyFunction_GET_GLOBALS(func), (PyObject *)NULL,
 &PyTuple_GET_ITEM(arg, 0), PyTuple_GET_SIZE(arg),
 k, nk, d, nd,
 PyFunction_GET_KW_DEFAULTS(func),
 PyFunction_GET_CLOSURE(func));
argcount, kwcount and defcount are int, whereas function_call() pass Py_ssize_t values.
function_call() should check PyTuple_GET_SIZE(arg) <= INT_MAX, nk <= INT_MAX and nd <= INT_MAX.

And another one:
..\Python\ceval.c(4271): warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
..\Python\ceval.c(4459): warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data [C:\buildbot.python.org3円.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
First in fast_function(), nd type is int:
 if (argdefs != NULL) {
 d = &PyTuple_GET_ITEM(argdefs, 0);
 ==> nd = Py_SIZE(argdefs); <=== HERE
 }
 return PyEval_EvalCodeEx((PyObject*)co, globals,
 (PyObject *)NULL, (*pp_stack)-n, na,
 (*pp_stack)-2*nk, nk, d, nd, kwdefs,
 PyFunction_GET_CLOSURE(func));
Second in ext_do_call(), nstar type is int:
 nstar = PyTuple_GET_SIZE(stararg);
Must check: Py_SIZE(argdefs) <= INT_MAX and PyTuple_GET_SIZE(stararg) <= INT_MAX.

I don't think they are actually the *same* issue.
For the limitations wrt. code objects (maximum size of byte code, maximum number of local variables, maximum number of parameters, etc.), I recommend the following thorough procedure:
1. document in a single text file all the limitations
2. check for each one whether an int is sufficient to represent them at runtime.
3. if yes: leave all structure definitions as-is. Local variables might be changed to size_t where this simplifies the code. Otherwise, Py_SAFE_DOWNCAST should be used where the actual value ought to be valid already. Runtime errors where a value may come from the outside that might be out of range.
4. if not: widen the structures to Py_ssize_t.

Here is a patch adding Py_SAFE_DOWNCAST(). For update_star_args(), I changed the type instead, because I prefer to avoid Py_SAFE_DOWNCAST() when possible.
Modify PyEval_EvalCodeEx() and PyCode_New() to use Py_ssize_t would be more correct, but it may be slower if Py_ssize_t is larger than int, and I hope that nobody calls functions with more than INT_MAX parameters! It would be completly inefficient!

Note this is referenced from #18407.

Many of these overflows can be provoked by specially constructed function, code object or bytecode.
Also I think following examples crash or return wrong result on 64 bit platform:
def f(*args, **kwargs): return len(args), len(kwargs)
f(*([0]*(2**32+1)))
f(**dict.fromkeys(map(hex, range(2**31+1))))
Here is updated patch which handles overflows in non-debug build. It prevent creating Python function with more than 255 default values (in any case compiler and interpreter don't support more than 255 arguments) and raise exception when function is called with too many arguments or too large *args or **kwargs.

Other than my one query on the review, code_ssize_t_2.patch.patch looks good to me.

It is possible to reproduce original bug without hacking the code object or bytecode:
>>> eval('lambda %s, *args, **kwargs: (lambda:args)' % (', '.join('a%d'%i for i in range(253)),))(*range(256))()
(253, 254, 255)
>>> eval('lambda %s, *args, **kwargs: (lambda:args)' % (', '.join('a%d'%i for i in range(254)),))(*range(256))()
(254, 255)
>>> eval('lambda %s, *args, **kwargs: (lambda:args)' % (', '.join('a%d'%i for i in range(255)),))(*range(256))()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<string>", line 1, in <lambda>
NameError: free variable 'args' referenced before assignment in enclosing scope
>>> eval('lambda %s, *args, **kwargs: (lambda:kwargs)' % (', '.join('a%d'%i for i in range(253)),))(*range(256))()
{}
>>> eval('lambda %s, *args, **kwargs: (lambda:kwargs)' % (', '.join('a%d'%i for i in range(254)),))(*range(256))()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<string>", line 1, in <lambda>
NameError: free variable 'kwargs' referenced before assignment in enclosing scope
>>> eval('lambda %s, *args, **kwargs: (lambda:kwargs)' % (', '.join('a%d'%i for i in range(255)),))(*range(256))()
0

Could we try and get this closed please, as I'm always a little concerned that a code change causes a genuine warning that should be actioned, but it gets masked by all the others.

New changeset e615718a6455 by Victor Stinner in branch 'default':
Use Py_ssize_t in _PyEval_EvalCodeWithName()
https://hg.python.org/cpython/rev/e615718a6455 

Since 3.7 the number of arguments no longer limited by 255 (see issue12844 and issue18896).