homepage

When I use the following configuration (in order to build a 32-bit Python a 64-bit OS):
 ./configure --with-computed-gotos CFLAGS=-m32 LDFLAGS=-m32
I then get two failures in test_distutils:
test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase) ... /tmp/tmpssz80G/tmp/tmpssz80G/xxmodule.o: could not read symbols: File in wrong format
collect2: ld a retourné 1 code d'état d'exécution
ERROR
[...]
test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... /tmp/tmphxwa1S/tempt/tmp/tmpuD17us/foo.o: could not read symbols: File in wrong format
collect2: ld a retourné 1 code d'état d'exécution
ERROR
[...]
======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 254, in link
 self.spawn(linker + ld_args)
 File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 909, in spawn
 spawn(cmd, dry_run=self.dry_run)
 File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 34, in spawn
 _spawn_posix(cmd, search_path, dry_run=dry_run)
 File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 138, in _spawn_posix
 % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/antoine/py3k/m32/Lib/distutils/tests/test_build_ext.py", line 65, in test_build_ext
 cmd.run()
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 347, in run
 self.build_extensions()
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 456, in build_extensions
 self.build_extension(ext)
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 543, in build_extension
 target_lang=language)
 File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 719, in link_shared_object
 extra_preargs, extra_postargs, build_temp, target_lang)
 File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 256, in link
 raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1
======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 254, in link
 self.spawn(linker + ld_args)
 File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 909, in spawn
 spawn(cmd, dry_run=self.dry_run)
 File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 34, in spawn
 _spawn_posix(cmd, search_path, dry_run=dry_run)
 File "/home/antoine/py3k/m32/Lib/distutils/spawn.py", line 138, in _spawn_posix
 % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/antoine/py3k/m32/Lib/distutils/tests/test_build_ext.py", line 321, in test_get_outputs
 cmd.run()
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 347, in run
 self.build_extensions()
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 456, in build_extensions
 self.build_extension(ext)
 File "/home/antoine/py3k/m32/Lib/distutils/command/build_ext.py", line 543, in build_extension
 target_lang=language)
 File "/home/antoine/py3k/m32/Lib/distutils/ccompiler.py", line 719, in link_shared_object
 extra_preargs, extra_postargs, build_temp, target_lang)
 File "/home/antoine/py3k/m32/Lib/distutils/unixccompiler.py", line 256, in link
 raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

It turns out, quite expectedly, that distutils doesn't reuse the various LDFLAGS recorded by sysconfig (why are there three of them?), but instead only uses LDSHARED:
>>> sysconfig.get_config_var('LDSHARED')
'gcc -pthread -shared'
>>> sysconfig.get_config_var('LDFLAGS')
'-m32 '
>>> sysconfig.get_config_var('PY_LDFLAGS')
'-m32'
>>> sysconfig.get_config_var('CONFIGURE_LDFLAGS')
'-m32'
I would suggest bumping this to critical or even release blocker, since building of extensions is broken when using non-default linker flags.

Following patch works here. I've added XXX tags in strange places.

Ok, there is also a failure in test_sysconfig (the patch makes no difference):
======================================================================
FAIL: test_ldshared_value (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/antoine/py3k/m32/Lib/test/test_sysconfig.py", line 285, in test_ldshared_value
 self.assertIn(ldflags, ldshared)
AssertionError: '-m32 ' not found in 'gcc -pthread -shared'

It appears that 2.7 has the same issue, although fewer tests fail (apparently not all 3.2 tests have been backported).

May be related to #4010 and #9047
The bug report that triggered the edits to configure is #9189 

Patch looks like it at least can't make things worse to me. linker_exe should probably also get LDFLAGS (but not LDSHARED).

It is however not important enough to block 3.2a2.

Other patch, fixes all failures.

I agree with Georg: The patch seems not to make things worse, and if it makes the tests pass, I think it should be committed. Antoine, you decide if you want to commit it now or wait for Tarek’s review.

> I agree with Georg: The patch seems not to make things worse, and if
> it makes the tests pass, I think it should be committed. Antoine, you
> decide if you want to commit it now or wait for Tarek’s review.
Well, ideally, someone should have knowledge of what is precisely needed
for these configuration variables. Otherwise the overall robustness of
distutils and friends won't really progress.

I agree. This expert is not me, and my reply was only pragmatic.
Martin: This is a bug about LDSHARED vs. LDFLAGS vs. PY_LDFLAGS vs. CONFIGURE_LDFLAGS. Could you maybe shed some light about the difference between these configure variables and review the small patch made by Antoine? Thanks.

Committed to 3.x in r85353.

Backported to 2.7 in r85358.

There are now failures on the FreeBSD 7.2 buildbot:
======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 254, in link
 self.spawn(linker + ld_args)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 909, in spawn
 spawn(cmd, dry_run=self.dry_run)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 34, in spawn
 _spawn_posix(cmd, search_path, dry_run=dry_run)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
 % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/tests/test_build_ext.py", line 65, in test_build_ext
 cmd.run()
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 347, in run
 self.build_extensions()
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
 self.build_extension(ext)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
 target_lang=language)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
 extra_preargs, extra_postargs, build_temp, target_lang)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 256, in link
 raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1
======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 254, in link
 self.spawn(linker + ld_args)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 909, in spawn
 spawn(cmd, dry_run=self.dry_run)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 34, in spawn
 _spawn_posix(cmd, search_path, dry_run=dry_run)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
 % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/tests/test_build_ext.py", line 321, in test_get_outputs
 cmd.run()
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 347, in run
 self.build_extensions()
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
 self.build_extension(ext)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
 target_lang=language)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
 extra_preargs, extra_postargs, build_temp, target_lang)
 File "/usr/home/db3l/buildarea/3.x.bolen-freebsd7/build/Lib/distutils/unixccompiler.py", line 256, in link
 raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1
Of course since distutils doesn't tell us what the gcc error message is, it's hard to do anything.

Apparently, the error message is:
gcc: ${LDFLAGS}: No such file or directory

Notice that the error shows further up in the output, at the point where the test was actually run:
gcc: ${LDFLAGS}: No such file or directory

The cause for the failure seems to be a bug in the dreaded _parse_makefile(). When you give it a Makefile such as:
LDFLAGS=-foo
PY_LDFLAGS=-bar
LDSHARED=$(CC) -shared ${LDFLAGS} $(PY_LDFLAGS)
It outputs the following variables:
{'CC': '',
 'LDFLAGS': '-foo',
 'LDSHARED': '-shared ${LDFLAGS} -bar',
 'PY_LDFLAGS': '-bar'}
That is, ${LDFLAGS} isn't expanded.
However, since the Makefile now explicitly appends PY_LDFLAGS to LDSHARED, we could also stop doing so in ./configure.

> However, since the Makefile now explicitly appends PY_LDFLAGS to
> LDSHARED, we could also stop doing so in ./configure.
I've committed a patch which does just this (in r85422). Hopefully it
won't break anything.

These changes cause some regressions:
- distutils builds third-party extensions with LDFLAGS, which were used to build CPython. (This is more important regression.)
 This problem concerns Python 2.7 and 3.2.
- distutils builds third-party extensions with LDFLAGS environment variable passed twice to compiler (distutils uses LDSHARED, which now contains $(LDFLAGS), and uses LDFLAGS).
 This problem concerns Python 3.2.
Example for 3.2:
x86_64-pc-linux-gnu-gcc -pthread -shared ${LDFLAGS_from_CPython} ${LDFLAGS_from_current_environment} ${LDFLAGS_from_current_environment} ${CFLAGS} ${object_files} -L/usr/lib64 -lpython3.2mu -o ${extension_module}

> These changes cause some regressions:
> - distutils builds third-party extensions with LDFLAGS, which were
> used to build CPython. (This is more important regression.)
> This problem concerns Python 2.7 and 3.2.
Well, is this a problem? This sounds like the expected behaviour to me.
Especially if Python was packaged was someone else and you don't want
the user to manually set LDFLAGS each time they run a setup.py.

> I've committed a patch which does just this (in r85422). Hopefully it
> won't break anything.
Could I convince you to patch 2.7 as well?

> > I've committed a patch which does just this (in r85422). Hopefully it
> > won't break anything.
> 
> Could I convince you to patch 2.7 as well?
It was backported in r86848. This means it should be available in 2.7.2.