homepage

The commit 1ec63b62035e73111e204a0e03b83503e1c58f2e of bpo-39763 broke the Python compilation on AIX:
https://bugs.python.org/issue39763#msg363749
--
The last successful build was before the commit 1ec63b62035e73111e204a0e03b83503e1c58f2e:
https://buildbot.python.org/all/#/builders/119/builds/383
_socket compilation:
(...) -o build/lib.aix-7100-9898-32-3.9-pydebug/_socket.so
pythoninfo:
sys.path: [
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
 '/home/buildbot/.local/lib/python3.9/site-packages']
Both steps use "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.
--
Recent failure:
https://buildbot.python.org/all/#/builders/119/builds/451
_socket compilation:
(...) -o build/lib.aix-7104-1806-32-3.9-pydebug/_socket.so
pythoninfo:
sys.path: [
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
 '/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
 '/home/buildbot/.local/lib/python3.9/site-packages']
So the compilation uses "build/lib.aix-7104-1806-32-3.9-pydebug/" directory, whereas pythoninfo uses "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.
It can explain why setup.py fails to import _socket later: it was built in a different directory.
--
I see that _aix_support._aix_bosmp64() has two code paths depending on the subprocess module can be imported:
# subprocess is not necessarily available early in the build process
# if not available, the config_vars are also definitely not available
# supply substitutes to bootstrap the build
try:
 import subprocess
 _have_subprocess = True
 _tmp_bd = get_config_var("AIX_BUILDDATE")
 _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
 _have_subprocess = False
 _tmp_bd = None
 _bgt = "powerpc-ibm-aix6.1.7.0"
def _aix_bosmp64():
 # type: () -> Tuple[str, int]
 """
 Return a Tuple[str, int] e.g., ['7.1.4.34', 1806]
 The fileset bos.mp64 is the AIX kernel. It's VRMF and builddate
 reflect the current ABI levels of the runtime environment.
 """
 if _have_subprocess:
 # We expect all AIX systems to have lslpp installed in this location
 out = subprocess.check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
 out = out.decode("utf-8").strip().split(":") # type: ignore
 # Use str() and int() to help mypy see types
 return str(out[2]), int(out[-1])
 else:
 from os import uname
 osname, host, release, version, machine = uname()
 return "{}.{}.0.0".format(version, release), _MISSING_BD
--
_aix_support._aix_bosmp64() is called by _aix_support.aix_platform() which is called by get_host_platform() of distutils.util.
Currently, setup.py does:
* Inject _bootsubprocess into sys.modules['subprocess'] so "import subprocess" works
* Build all C extensions
* Remove sys.modules['subprocess'], so the next "import subprocess" may or may not load Lib/subprocess.py which uses the newly built C extensions like _posixsubprocess and select
* Attempt to load C extensions: if an import fails, rename the file: it happens for _asyncio on AIX, that's the bug

AIX_BUILDDATE variable is written in pyconfig.h by configure:
> https://buildbot.python.org/all/#/builders/119/builds/383
configure: "checking for the system builddate... 1806"
> https://buildbot.python.org/all/#/builders/119/builds/451
configure: "checking for the system builddate... 1806"
So AIX_BUILDDATE seems to be set in both case.
But on build 451, the "build/lib.aix-7100-9898-32-3.9-pydebug" directory name contains 9898: this number comes from _aix_support._MISSING_BD. It's used when _have_subprocess is false or when get_config_var("AIX_BUILDDATE") cannot be converted into an integer.

While looking through the history of bot builds - I consistently see 420 tests being done on one bot - but test_socket does not always show up in the list.
I'll look at this as I can, but "free-time" is limited. Delay is not a lack of interest.

Michael.Felt: Can you please try to build the master branch of Python with the attached setup-aix.patch applied?
Something like:
git checkout master
git apply setup-aix.patch
./configure CFLAGS="-O0"
make 2>&1|tee make.log
Then attach make.log.

Another approach to patch _aix_support.py, replace:
try:
 import subprocess
 _have_subprocess = True
 _tmp_bd = 
 _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
 _have_subprocess = False
 _tmp_bd = None
 _bgt = "powerpc-ibm-aix6.1.7.0"
with:
try:
 import subprocess
 _have_subprocess = True
 _tmp_bd = get_config_var("AIX_BUILDDATE")
 _bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
 import _bootsubprocess as subprocess
 _have_subprocess = True
 _tmp_bd = None
 _bgt = "powerpc-ibm-aix6.1.7.0"
Use _bootsubprocess when subprocess is not available. If this approach works, _have_subprocess=False code path can be removed.

I tried your patch on AIX 7.2, looks like you need to shift sys import to the top
Traceback (most recent call last):
 File "/home/isidentical/cpython/./setup.py", line 16, in <module>
 sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: The error code from the last command is 1.
Stop.

Yeah, sys must be imported first. Did you try to fix the patch? Does it work?

I'll take a look at what you are suggesting.
The starting point (before the rm command) is the make command that I run again.
What I notice - read am thinking - is that _socket.so is being created by the "setup.py build" - so, if you can help me make that bit more verbose, I'll work on that too.
+++
Script command is started on Wed Mar 11 15:29:44 CDT 2020.
$
$
$
$ rm ./build/lib.aix-7200-1543-32-3.9/_socket.so ./build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
$
$
$ make V=1
 CC='gcc -pthread' LDSHARED='Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
ldd: /lib/libreadline.a: File is an archive.
building '_asyncio' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
 -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_asynciomodule.c -o build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o -o build/lib.aix-7200-1543-32-3.9/_asyncio.so
building '_socket' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
 -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/socketmodule.c -o build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o -o build/lib.aix-7200-1543-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Following modules built successfully but were removed because they could not be imported:
_asyncio

So, with the patch - the process stops at:
aixtools@x064:[/home/aixtools/python-3.9]make
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
Traceback (most recent call last):
 File "/home/aixtools/python-3.9/Lib/subprocess.py", line 72, in <module>
 import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/aixtools/python-3.9/./setup.py", line 4, in <module>
 import subprocess
 File "/home/aixtools/python-3.9/Lib/subprocess.py", line 77, in <module>
 import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/aixtools/python-3.9/./setup.py", line 16, in <module>
 sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: 1254-004 The error code from the last command is 1.
Stop.

So, this is what I have on screen. Will add the log in a moment.
[1] + Done make 2>&1|tee make.log &
aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/__pycache__/socket.cpython-39.opt-1.pyc
./Lib/__pycache__/socket.cpython-39.opt-2.pyc
./Lib/__pycache__/socket.cpython-39.opt-4.pyc
./Lib/__pycache__/socket.cpython-39.pyc
./Lib/__pycache__/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/__pycache__/mock_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7104-1806-32-3.9/_socket.so
./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
aixtools@x064:[/home/aixtools/python-3.9]rm ./build/lib.aix-7104-1806-32-3.9/_socket.so ./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
aixtools@x064:[/home/aixtools/python-3.9]make V=1
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7104-1806-32-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
 1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7104-1806-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm _lzma _sqlite3
_tkinter readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Following modules built successfully but were removed because they could not be imported:
_asyncio

OK. I removed the _aix_support.py from the formula.
After make see the new (rather) old build paths for "socket"
aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/__pycache__/socket.cpython-39.opt-1.pyc
./Lib/__pycache__/socket.cpython-39.opt-2.pyc
./Lib/__pycache__/socket.cpython-39.opt-4.pyc
./Lib/__pycache__/socket.cpython-39.pyc
./Lib/__pycache__/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/__pycache__/mock_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socket.cpython-39.pyc
./Lib/test/__pycache__/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7.1-3.9/_socket.so
./build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
I remove these two files again and run make:
aixtools@x064:[/home/aixtools/python-3.9]make
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7.1-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
 1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7.1-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm _lzma _sqlite3
_tkinter readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Following modules built successfully but were removed because they could not be imported:
_asyncio
The diff!
diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
 _tmp_bd = get_config_var("AIX_BUILDDATE")
 _bgt = get_config_var("BUILD_GNU_TYPE")
 except ImportError: # pragma: no cover
- _have_subprocess = False
+ import _bootsubprocess as subprocess
+ _have_subprocess = True
 _tmp_bd = None
 _bgt = "powerpc-ibm-aix6.1.7.0"
diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
 machine += ".%s" % bitness[sys.maxsize]
 # fall through to standard osname-release-machine representation
 elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
 elif osname[:6] == "cygwin":
 osname = "cygwin"
 rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
 machine += ".%s" % bitness[sys.maxsize]
 # fall through to standard osname-release-machine representation
 elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
 elif osname[:6] == "cygwin":
 osname = "cygwin"
 import re
IMHO - this problem has nothing to do with _aix_support.py - That issue was fixed earlier by the new _bootsubprocess.

Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make.
> NameError: name 'sys' is not defined
Right, that's a stupid bug in my patch. You must import sys before.

Try to replace Lib/_aix_support.py with attached _aix_support.py: in short, it uses _bootsubprocess if subprocess is not available.

Actually, I had already done that:
diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
 _tmp_bd = get_config_var("AIX_BUILDDATE")
 _bgt = get_config_var("BUILD_GNU_TYPE")
 except ImportError: # pragma: no cover
- _have_subprocess = False
+ import _bootsubprocess as subprocess
+ _have_subprocess = True
 _tmp_bd = None
 _bgt = "powerpc-ibm-aix6.1.7.0"
And after that test, I tried with no calls to _aix_support.py
diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
 machine += ".%s" % bitness[sys.maxsize]
 # fall through to standard osname-release-machine representation
 elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
 elif osname[:6] == "cygwin":
 osname = "cygwin"
 rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
 machine += ".%s" % bitness[sys.maxsize]
 # fall through to standard osname-release-machine representation
 elif osname[:3] == "aix":
- from _aix_support import aix_platform
- return aix_platform()
+ return "%s-%s.%s" % (osname, version, release)
+ # from _aix_support import aix_platform
+ # return aix_platform()
 elif osname[:6] == "cygwin":
 osname = "cygwin"
 import re
±±±±±±±±
So, even when _aix_support.py is not involved at all, _socket.so is being ignored - even though it was built.

re: Michael: this issue is about bootstraping Python. If you want to test a patch or test something else, you must restart from a clean copy of the source code. Either use "make distclean", "git clean -fdx", or recreate the source directory (git clone, decompress an archive, etc.). Then restart from scratch: ./configure && make.
Understood: between these two timestamps: Date: 2020年03月11日 18:55 and Date: 2020年03月11日 21:36 - I was doing just that on 4 different VM's: AIX 5.3, 6.1, 7.1 and 7.2 - just in case there may be anything specific re: AIX versions (the bots run on AIX 7.1 and 7.2). I also switched away from any NFS based builds because test behaves differently when the build area is an NFS area compared to jfs2.
FYI: I have decided to focus on AIX 7.1 and AIX 7.2 - and am only using gcc (I was also testing runs with both xlc and gcc before I started posting).
Further, I like your suggestion to try: subprocess except _bootsubprocess and shall make a PR to that end. Much more clear about what is going on.
What I am going to do bewtween now and my next post is again - clean slate - update master to latest version - on two servers.
On one server I'll apply your suggestion to _aix_platform.py; on the other I'll patch Lib/distutils/util.py and Lib/sysconfig.py to not even call _aix_platform.py. Ideally, there will be a difference that leads to an understanding of the root cause.

The good news! Your patch, better rewrite, of _aix_platform.py is working!
Many thanks!

I converted attached _aix_support.py into PR 18970 (with minor changes).

New changeset c846ef004d79ee8e9645d3e5e8b3b0cb97b5013f by Victor Stinner in branch 'master':
bpo-39936: _aix_support uses _bootsubprocess (GH-18970)
https://github.com/python/cpython/commit/c846ef004d79ee8e9645d3e5e8b3b0cb97b5013f

Fantastic! Many thanks!

POWER6 AIX 3.x and PPC64 AIX 3.x buildbots are back to green.