homepage

[This bug was discovered by Alexander Belopolsky, during the work on Issue 9317]
Bug report
**********
The attached traceme.py file demonstrates the following problem:
With python 2.7:
$ python2 -m trace -c -s traceme.py
lines cov% module (path)
 1 100% threading (Lib/threading.py)
 6 100% traceme (traceme.py)
The first entry is clearly spurious, but traceme looks right. With py3k, however, I get
$ python3 -m trace -c -s traceme.py
lines cov% module (path)
 1 100% threading (Lib/threading.py)
No traceme line at all.
Analysis
********
When trace.py actually runs the script, it uses this code in 3.x:
 t.run('exec(%r)' % (script,))
instead of this code in 2.x:
 t.run('execfile(%r)' % (progname,))
`exec` doesn't have the program name to attach to the code object created, and calls it '<string>' by default. However, this file name is ignored by the trace module.
`execfile` doesn't exist in 3.x, so an alternative approach is needed.
Proposed solution
*****************
Instead of t.run as displayed above, use this code:
 with open(progname) as fp:
 code = compile(fp.read(), progname, 'exec')
 t.run(code)
The code object created by `compile` attaches the program name and is executable by t.run (because t.run actually calls `exec` which can run the result of `compile`).
This solution fixes the problem:
$ py3d -m trace -c -s traceme.py 
lines cov% module (path)
 1 100% threading (/home/eliben/python_src/eliben-py3k/Lib/threading.py)
 6 100% traceme (traceme.py)
I'll attach a patch file that was creates vs. the latest trunk, with this fix.

[Copying msg111059 from issue9317]
On Wed, Jul 21, 2010 at 9:45 AM, Eli Bendersky <report@bugs.python.org> wrote:
..
> As far as I understand, when you run:
>
> py3d -m trace -C pickle-trace.d -c -m test_pickle.py
>
> The first -m flag applies to the trace module. Python uses
> runpy.run_module to load it, and then passes it its arguments (-C and
> onwards).
>
When I said "run from command line", I meant something like
$ python Lib/test/test_pickle.py
I am not sure if this uses runpy machinery, but I suspect it does.
 The trace output should not include runpy calls that load the trace
module itself, but
$ python -m trace <flags> Lib/test/test_pickle.py
might show the runpy calls that occur when you run
$ python Lib/test/test_pickle.py
without tracing. BTW, too bad that '-m' is taken, but I think
trace.py should grow a --run-module option so that you can trace
equivalents of python -m executions:
$ python -m trace <flags> --run-module test.test_pickle
> Moreover, since runpy is only the machinery to execute trace.py, does
> it even make sense to include it in coverage?
>
> What do you think?
I think you should ask python-dev. :-) Note that most C/C++ coverage
tools do include pre-main calls in coverage. In case of C++, a lot
of things may happen before main().
Whatever the solution is, it should be unified between trace, profile,
and pdb modules.

Committed with a minor change. (There is no need to keep the source file open during the trace run.) See r83035 (r83037 in release31-maint.)
I will open a separate issue to discuss a possible RFE related to msg111074.

Reopening because I think this fix is responsible for a new problem:
$ ./python.exe -m trace -C. -s -c Lib/test/regrtest.py 
Traceback (most recent call last):
 ...
 File "Lib/test/regrtest.py", line 1458, in <module>
 assert __file__ == os.path.abspath(sys.argv[0])
AssertionError
[98796 refs]

It appears to affect 2.7 as well (probably because of a back-port from py3k to 27-maint in revision 83421).

Looking at Lib/test/regrtest.py:
 if __name__ == '__main__':
 # Simplification for findtestdir().
 assert __file__ == os.path.abspath(sys.argv[0])
I'm not sure regrtest.py is prepared to be run from trace at all. Note that this error also happens when running 'python -m profile regrtest.py', so it's rather regrtest.py specific than trace.py specific.
Anyway, trace.py itself currently passes the cmd-line argument as __file__ and not its full path, which causes the breakage in regrtest.py. The full path *could* be easily passed, but I think we should address this problem from regrtest.py's side.

> The full path *could* be easily passed, but I think we should
> address this problem from regrtest.py's side.
I disagree. There may be other scripts that rely on __main__.__file__ being an absolute path and we cannot require everyone to change their assumptions to make their scripts traceable.
It looks like absolute path assumption was added by Florent in r78723 (r78719 in trunk). Florent, I would like to hear from you before deciding how to proceed. See also issue #7712.

I've made the wrong assumption that test.regrtest.__file__ is always absolute.
Here is a patch which should fix it.

regrtest fixed with r84776 (3.x) and r84779 (2.7)

Thanks, Florent.