homepage

Attached patch implements the largest part of the PEP 511:
* add sys.get_code_transformers() and sys.set_code_transformers()
* add sys.implementation.optim_tag
* add -o command line option setting sys.implementation.optim_tag
* modify importlib to use sys.implementation.optim_tag to build .pyc filenames
* modify importlib to raise an ImportError when the .pyc file is missing, the .py is available, and the optimizer tag created from code transformers is different than the current sys.implementation.optim_tag
* add Lib/test/test_pep511.py: test changes especially bytecode and AST transformers
* add a PeepholeOptimizer object created by default for sys.set_code_transformers()
* skip the peephole optimizer when -o noopt is used
* add sys field to the PyInterpreterState structure: used by _PySys_GetCodeTransformers() to get code transformers
* update unit tests to use the optimizer to create the .pyc filename
An optimizer tag must contain characters: '.', '-', directory separators or NUL character.
Brett Canon proposed to not include the optimizer tag in .pyc filenames if it is "opt" to limit backward incompatible changes.
Note: The PEP 511 is under discussion on python-ideas. The patch implements the first version of the PEP, sent to python-ideas, so it doesn't include proposed changes like changing code_transformer() prototype to "def code_transformer(self, code, context)".
See also the issue #26100: add test.support.optim_args_from_interpreter_flags().

Missing patch for test_sys testing the new sys.get/set_code_transformers() functions.

main.c:
+ case 'o':
+ if (wcslen(_PyOS_optarg) == 0
+ || wcschr(_PyOS_optarg, L'.')
+ || wcschr(_PyOS_optarg, L'-')
...
'-' character must be valid for -o argument.

sys_init_code_transformer: Py_DECREF(name); must be Py_XDECREF(name); at exit, it can NULL. An unit test on an invalid transformer (with no name) is missing?

Patch version 2:
* allow "-" character in the argument of the -o command line option
* add an unit test on invalid type for code transformer
* fix a crash in sys.set_code_transformers(): replace Py_DECREF() with Py_XDECREF()

(oops, i posted the same message with patch twice)

Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

Rebased patch.

Patch version 4: Fix sys_set_code_transformers(), initialize seq to NULL to fix a crash on error handling.

> Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.
Additionnal pycf_transformed_ast.patch implements it. The patch should be applied on top of transformers-4.patch.
Note: PyCF_TRANSFORMED_AST has the same value (0x1000) than the old constant CO_GENERATOR_ALLOWED. This constant was removed in Python 2.5.0 by the changeset 6b42920accc9 in 2006. Is it an issue?

Rebased patch combining also pycf_transformed_ast.patch.

Patch version 6: updated to the new code_transformer() API, it now takes a code object as input and must return a new code object.
Sadly, it looks like there are reference leaks (try: ./python -m test -R 3:3 test_import).

TODO: implement a fast-path for PyCode_Optimize() avoiding completly the "expensive" code transformer API, before the first call to sys.set_code_transformer().
It would avoid list <=> tuple conversions for code constants, and the need of creating two code objects.
In practice, this fast-path should be taken in most cases.

Recently, some people asked me for an update for my FAT Python project. So I rebased this change I wrote 1 year 1/2 and adapted it for the new code base:
* I renamed test_pep511.py to test_code_transformer.py
* I removed the sys module from the PyInterpreterState structure (added in my previous patch), since Eric Snow and Nick Coghlan removed a variant of the sys module from this structure for their work on subinterpreters and reworked Python initialization
The PEP 511 is not accepted, so the implementation is still a work-in-progress (WIP) and must not be merged.

I rejected my own PEP 511, so I now reject this issue as well.
https://mail.python.org/pipermail/python-dev/2017-October/149903.html