homepage

The distutils commands 'build_scripts' and 'install_data' both may end
up installing .py files. Such .py file should be able tobe run through
lib2to3 in the same way supported by build_py.
pywin32 has ended up with something like:
 class my_build_scripts(build_scripts):
 def copy_file(self, src, dest):
 dest, copied = build_scripts.copy_file(self, src, dest)
 # 2to3
 if not self.dry_run and copied:
 refactor_filenames([dest])
 return dest, copied
where 'refactor_filenames' is (basically) the lib2to3 block from
build_py moved to a utility function.

Here is a patch. It refactors the 2to3 support into a function in util,
and adds build_scripts support.
For build_scripts, the approach is slightly different from the one
proposed by Mark: I modify copy_scripts to collect the modified files,
and return them along with outfiles. Modifying copy_file is
insufficient, as it doesn't support the case of adjust.
I'm skeptical about adding build_data support, as it's not obvious what
files would need conversion. Users should adjust their build_data
commands to invoke distutils.util.run_2to3. I would appreciate a
confirmation that this function has a useful API.
In this version, I don't pass any options to the RefactoringTool
anymore, and I arrange to delete the backup files.

If you'd like, I can add support for skipping backup files to 2to3.

> If you'd like, I can add support for skipping backup files to 2to3.
That would be useful, I think.

Ok. I've refactored RefactoringTool so that the write_file method
doesn't make a backup file by default.

> I'm skeptical about adding build_data support, as it's not 
> obvious what files would need conversion.
All .py files should be converted. I can't think why a project would
use this 2to3 capability for scripts and packages, but not .py files in
data files. pywin32 uses data files to install 'demo' scripts.
Would the motivation become clearer if we modified
Demo/distutils/test2to3 to install a README.txt and a demo .py file
(where the demo .py file should not be installed in the global 'scripts'
directory, but instead in a 'demos' sub-directory of the package)?
> Users should adjust their build_data commands to invoke
> distutils.util.run_2to3.
True - but the same justification could be used to avoid adding support
to build_scripts. I guess the question is 'how common is it that .py
files will be included in data_files?' My experience is that it is
common enough to warrant support (but obviously others experiences may
be different)
> I would appreciate a confirmation that this function has a useful API.
I haven't actually applied the patch and modified pywin32 to use it, but
the API certainly looks useful from reading the patch. +1
Thanks.

Thinking more about data_files, I'd agree that blindly converting all
.py files and nothing more isn't as useful for install_data as the other
commands. It might make more sense to allow data_files to specify a
list of patterns that will be matched to perform the conversion, with
the default being all .py files.

> All .py files should be converted.
I disagree. For example, people decided that all .py files in Tools
should get converted, converting Tools/msi in the process. Now,
Tools/msi is intended to be run in an *older* Python version than the
one being packaged; I currently use 2.4 to do the packaging. It took me
some time to undo this conversion.
It's convenient if users can easily express which files they want to
convert, but it must be a selective, explicit choice.
> I can't think why a project would
> use this 2to3 capability for scripts and packages, but not .py files in
> data files.
See above.
> Would the motivation become clearer if we modified
> Demo/distutils/test2to3 to install a README.txt and a demo .py file
> (where the demo .py file should not be installed in the global 'scripts'
> directory, but instead in a 'demos' sub-directory of the package)?
I understand the motivation fully (I think). I just question whether it
is general. Packages that have this need (to 2to3-convert data files)
would have to come up with their own command subclass still, which
hopefully is very easy with what we provide.
IMO, this is a case where we should err on the conservative side. If we
don't provide the command in 3.0, people will write their own. If your
assumption is correct that this is a common need, we can still add the
command into 3.1.

I'd like to see this in 3.0. It will help people porting to 3.0 better.

Review:
1. Since RefactoringTool no longer writes backup files, write_file
doesn't need to be overridden.
2. Don't worry about **kwds on log_error. It's not used at the moment.
3. It might be nice to let users pick which fixers they do or do not
want. Using only needed fixers can boost performance immensely.

Thanks for the review. Here is a revised version, addressing comments 1
and 2.
Comment 3 is address by exposing all arguments to RefactoringTool as
class variables in a new class Mixin2to3, from which build_py and
build_scripts inherit.

Ok. Looks good.

Committed as r67464.