homepage

I recently upgraded Python 3.6.0rc1 over 3.6.0b4. Python would invoke just fine after the update, but following a system restart, I'm experiencing crashes on any invocation of Python. I suspect it's just a bad/corrupted install. I will do more investigation, but here's the report from MacOS.
Process: Python [3211]
Path: /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 3.6.0rc1 (3.6.0rc1)
Code Type: X86-64 (Native)
Parent Process: zsh [990]
Responsible: Python [3211]
User ID: 501
Date/Time: 2016年12月12日 09:36:50.115 -0500
OS Version: Mac OS X 10.12.1 (16B2657)
Report Version: 12
Anonymous UUID: 31F72BE9-33AC-23C5-6CC1-DA2C20DDA3B2
Time Awake Since Boot: 1400 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
abort() called
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 	0x00007fffc65e0dda __pthread_kill + 10
1 libsystem_pthread.dylib 	0x00007fffc66cc787 pthread_kill + 90
2 libsystem_c.dylib 	0x00007fffc6546420 abort + 129
3 org.python.python 	0x00000001001280d1 Py_FatalError + 65
4 org.python.python 	0x0000000100129e21 _Py_InitializeEx_Private + 1377
5 org.python.python 	0x0000000100146c60 Py_Main + 1376
6 org.python.python 	0x0000000100000dfe 0x100000000 + 3582
7 org.python.python 	0x0000000100000c34 0x100000000 + 3124
Thread 0 crashed with X86 Thread State (64-bit):
 rax: 0x0000000000000000 rbx: 0x0000000000000006 rcx: 0x00007fff5bfff448 rdx: 0x0000000000000000
 rdi: 0x0000000000000307 rsi: 0x0000000000000006 rbp: 0x00007fff5bfff470 rsp: 0x00007fff5bfff448
 r8: 0x0000000000000000 r9: 0x0000000000000006 r10: 0x0000000008000000 r11: 0x0000000000000206
 r12: 0x00007fffcf286a20 r13: 0x0000000000000002 r14: 0x00007fffcf2a23c0 r15: 0x00007fff5bfff4d0
 rip: 0x00007fffc65e0dda rfl: 0x0000000000000206 cr2: 0x00007fffcf284128
 
Logical CPU: 0
Error Code: 0x02000148
Trap Number: 133
Binary Images:
 0x100000000 - 0x100000ff7 +org.python.python (3.6.0rc1 - 3.6.0rc1) <581ACC12-D39B-9ED6-E4C0-DC4709F9A849> /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
 0x100003000 - 0x100225ff7 +org.python.python (3.6.0rc1, [c] 2001-2016 Python Software Foundation. - 3.6.0rc1) <57D55331-6C7E-012A-47A2-5CA85BB2ECAC> /Library/Frameworks/Python.framework/Versions/3.6/Python
 0x7fff60b9d000 - 0x7fff60bda287 dyld (421.2) <F5F79617-5669-3424-A2CF-18B23BB0C0C7> /usr/lib/dyld
 0x7fffb11c3000 - 0x7fffb165cff7 com.apple.CoreFoundation (6.9 - 1348.15) <C25C6FB8-E934-349C-952C-26CF913BAC33> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
 0x7fffc4cdc000 - 0x7fffc4cddff3 libDiagnosticMessagesClient.dylib (102) <422911A4-E273-3E88-BFC4-DF6470E48242> /usr/lib/libDiagnosticMessagesClient.dylib
 0x7fffc4f1b000 - 0x7fffc4f1cff3 libSystem.B.dylib (1238) <F62ED8CF-37FB-30E3-A453-C62BA28CAA07> /usr/lib/libSystem.B.dylib
 0x7fffc502c000 - 0x7fffc502cff3 libauto.dylib (187) <5BBF6A00-CC76-389D-84E7-CA88EDADE683> /usr/lib/libauto.dylib
 0x7fffc504d000 - 0x7fffc50a3ff7 libc++.1.dylib (307.4) <BEE86868-F831-384C-919E-2B286ACFE87C> /usr/lib/libc++.1.dylib
 0x7fffc50a4000 - 0x7fffc50cefff libc++abi.dylib (307.2) <1CEF8ABB-7E6D-3C2F-8E0A-E7884478DD23> /usr/lib/libc++abi.dylib
 0x7fffc560a000 - 0x7fffc582ffff libicucore.A.dylib (57132.0.1) <063DBF01-CEF8-3C2E-9917-A08358BB006D> /usr/lib/libicucore.A.dylib
 0x7fffc5bbf000 - 0x7fffc5f8fd97 libobjc.A.dylib (706) <F9AFE665-A3A2-3285-9495-19803A565861> /usr/lib/libobjc.A.dylib
 0x7fffc63b6000 - 0x7fffc63c7ff3 libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib
 0x7fffc63d6000 - 0x7fffc63daff7 libcache.dylib (79) <84E55656-FDA9-3B29-9E4F-BE31B2C0AA3C> /usr/lib/system/libcache.dylib
 0x7fffc63db000 - 0x7fffc63e5fff libcommonCrypto.dylib (60092.20.1) <D2711CB2-2D9A-3238-8082-CDB8AE60017F> /usr/lib/system/libcommonCrypto.dylib
 0x7fffc63e6000 - 0x7fffc63edfff libcompiler_rt.dylib (62) <486BDE52-81B4-3446-BD72-23977CAE556F> /usr/lib/system/libcompiler_rt.dylib
 0x7fffc63ee000 - 0x7fffc63f6fff libcopyfile.dylib (138) <0DA49B77-56EC-362D-98FF-FA78CFD986D6> /usr/lib/system/libcopyfile.dylib
 0x7fffc63f7000 - 0x7fffc6479fdb libcorecrypto.dylib (442.20.2) <9846F683-6CED-3CE5-AE8A-B0A681F7FEEF> /usr/lib/system/libcorecrypto.dylib
 0x7fffc647a000 - 0x7fffc64acfff libdispatch.dylib (703.20.1) <FDC0F71D-A415-306A-ABE8-CA21E25A55BB> /usr/lib/system/libdispatch.dylib
 0x7fffc64ad000 - 0x7fffc64b2ff3 libdyld.dylib (421.2) <7BFA3476-6210-3BCB-8CE8-9B952F87BD84> /usr/lib/system/libdyld.dylib
 0x7fffc64b3000 - 0x7fffc64b3ffb libkeymgr.dylib (28) <09CD7CA6-46D2-3A9F-B9F1-7C4CA5CA0D68> /usr/lib/system/libkeymgr.dylib
 0x7fffc64c1000 - 0x7fffc64c1fff liblaunch.dylib (972.20.3) <7AB2E2EA-8B47-3420-87CE-5EE18A4EEE49> /usr/lib/system/liblaunch.dylib
 0x7fffc64c2000 - 0x7fffc64c7fff libmacho.dylib (894) <1EAE5ADD-490C-3B1F-9F97-447BA8E0E90F> /usr/lib/system/libmacho.dylib
 0x7fffc64c8000 - 0x7fffc64caff3 libquarantine.dylib (85) <78EF62D8-C890-3FC0-937A-C2FD8CEF8992> /usr/lib/system/libquarantine.dylib
 0x7fffc64cb000 - 0x7fffc64ccffb libremovefile.dylib (45) <C4FC07FF-ED86-382E-B06F-33C34718080C> /usr/lib/system/libremovefile.dylib
 0x7fffc64cd000 - 0x7fffc64e5ff7 libsystem_asl.dylib (349.1.1) <2217DE86-0635-393E-93DD-A1344EC0EF4B> /usr/lib/system/libsystem_asl.dylib
 0x7fffc64e6000 - 0x7fffc64e6ff7 libsystem_blocks.dylib (67) <B8C3701D-5A91-3D35-999D-2DC8D5393525> /usr/lib/system/libsystem_blocks.dylib
 0x7fffc64e7000 - 0x7fffc6574fef libsystem_c.dylib (1158204) <FB1702C3-C521-37BD-91CB-559168785D22> /usr/lib/system/libsystem_c.dylib
 0x7fffc6575000 - 0x7fffc6578ffb libsystem_configuration.dylib (888.20.5) <D7FC38C6-794C-3C26-AE4F-020A72E1BB79> /usr/lib/system/libsystem_configuration.dylib
 0x7fffc6579000 - 0x7fffc657cfff libsystem_coreservices.dylib (41.2) <ADE57FD1-AFEB-3E86-820D-E4F28330F76A> /usr/lib/system/libsystem_coreservices.dylib
 0x7fffc657d000 - 0x7fffc6595ffb libsystem_coretls.dylib (121.1.1) <8F7E9B12-400D-3276-A9C5-4546B0258554> /usr/lib/system/libsystem_coretls.dylib
 0x7fffc6596000 - 0x7fffc659cfff libsystem_dnssd.dylib (765.20.4) <49E37AB9-776C-3F84-8AC2-9BDD0438ED9D> /usr/lib/system/libsystem_dnssd.dylib
 0x7fffc659d000 - 0x7fffc65c6fff libsystem_info.dylib (503) <C686B834-5E7D-382C-AF6E-44AB78EE83E2> /usr/lib/system/libsystem_info.dylib
 0x7fffc65c7000 - 0x7fffc65e9ff7 libsystem_kernel.dylib (3789214) <651F249F-D797-3FBD-A66F-66DFB48FCF38> /usr/lib/system/libsystem_kernel.dylib
 0x7fffc65ea000 - 0x7fffc6631fe7 libsystem_m.dylib (3121.4) <E3370D16-EBAA-3C7F-AC56-2D6EAD7DB0A4> /usr/lib/system/libsystem_m.dylib
 0x7fffc6632000 - 0x7fffc6650ff7 libsystem_malloc.dylib (116) <3DD17B88-B7A4-38B9-9E95-AB88E1C3B647> /usr/lib/system/libsystem_malloc.dylib
 0x7fffc6651000 - 0x7fffc66a8ff3 libsystem_network.dylib (856.20.4) <0F16BBE0-3092-390E-96D9-0B5DCC97B317> /usr/lib/system/libsystem_network.dylib
 0x7fffc66a9000 - 0x7fffc66b2ff3 libsystem_networkextension.dylib (563.20.3) <C8C8807B-F5C1-307E-8D67-02793B62E9ED> /usr/lib/system/libsystem_networkextension.dylib
 0x7fffc66b3000 - 0x7fffc66bcff3 libsystem_notify.dylib (165.20.1) <D38E3A3C-AEEE-321A-861E-160ABE4B20B4> /usr/lib/system/libsystem_notify.dylib
 0x7fffc66bd000 - 0x7fffc66c5fe7 libsystem_platform.dylib (126.1.2) <884DDF42-3CAE-334A-82CE-965617130FB1> /usr/lib/system/libsystem_platform.dylib
 0x7fffc66c6000 - 0x7fffc66d0ff7 libsystem_pthread.dylib (218.20.1) <F35891AC-CA82-3F97-9BE6-15EC3B4C36F6> /usr/lib/system/libsystem_pthread.dylib
 0x7fffc66d1000 - 0x7fffc66d4ff7 libsystem_sandbox.dylib (592.21.2) <FCF7F7C2-79B5-321F-901D-A190CD3D22A2> /usr/lib/system/libsystem_sandbox.dylib
 0x7fffc66d5000 - 0x7fffc66d6fff libsystem_secinit.dylib (24) <A54B8FEF-E792-3C54-8E0B-E80A376662F2> /usr/lib/system/libsystem_secinit.dylib
 0x7fffc66d7000 - 0x7fffc66defff libsystem_symptoms.dylib (532.1.1) <B26F656E-94F9-3834-9B03-51C4FF11D1BF> /usr/lib/system/libsystem_symptoms.dylib
 0x7fffc66df000 - 0x7fffc66ffff7 libsystem_trace.dylib (518.20.8) <046443DA-0474-3A73-A69F-1DC21C49B8AD> /usr/lib/system/libsystem_trace.dylib
 0x7fffc6700000 - 0x7fffc6705ffb libunwind.dylib (35.3) <9F7C2AD8-A9A7-3DE4-828D-B0F0F166AAA0> /usr/lib/system/libunwind.dylib
 0x7fffc6706000 - 0x7fffc672fff7 libxpc.dylib (972.20.3) <CE86F4BA-EF31-33BD-8AC3-7BF25F632FA9> /usr/lib/system/libxpc.dylib
External Modification Summary:
 Calls made by other processes targeting this process:
 task_for_pid: 0
 thread_create: 0
 thread_set_state: 0
 Calls made by this process:
 task_for_pid: 0
 thread_create: 0
 thread_set_state: 0
 Calls made by all processes on this machine:
 task_for_pid: 1016
 thread_create: 0
 thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=130.3M resident=0K(0%) swapped_out_or_unallocated=130.3M(100%)
Writable regions: Total=35.7M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=35.7M(100%)
 
 VIRTUAL REGION 
REGION TYPE SIZE COUNT (non-coalesced) 
=========== ======= ======= 
Kernel Alloc Once 8K 2 
MALLOC 19.3M 11 
MALLOC guard page 16K 4 
VM_ALLOCATE 4K 2 
__DATA 3452K 49 
__LINKEDIT 113.4M 5 
__TEXT 16.8M 47 
__UNICODE 556K 2 
__UNIXSTACK 16.0M 2 
shared memory 8K 3 
=========== ======= ======= 
TOTAL 169.5M 117 
Model: MacBookPro13,2, BootROM MBP132.0226.B00, 2 processors, Intel Core i7, 3.3 GHz, 16 GB, SMC 2.37f18
Graphics: kHW_IntelIrisGraphics550Item, Intel Iris Graphics 550, Built-In
Memory Module: BANK 0/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320
Memory Module: BANK 1/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x157), Broadcom BCM43xx 1.0 (7.21.171.47.1a8)
Bluetooth: Version 5.0.1f7, 3 services, 27 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
PCI Card: pci1b21,1142, USB eXtensible Host Controller, Thunderbolt@195,0,0
USB Device: USB 3.0 Bus
USB Device: iBridge
USB Device: USB 3.0 Bus
USB Device: ASM107x
USB Device: TUSB3410 Boot Device
USB Device: Microsoft® LifeCam Cinema(TM)
USB Device: ASM107x
Thunderbolt Bus: MacBook Pro, Apple Inc., 11.9
Thunderbolt Bus: MacBook Pro, Apple Inc., 11.9
Thunderbolt Device: 34UM95, LG Electronics, 3, 24.1

A simple re-install corrected the issue and survived a restart. I suspect one or more components weren't properly upgraded or were corrupted for unrelated reasons. I suspect that it was an environmental issue and not an issue with the package itself.

Spontaneously, this issue has arisen again. Without a system restart or any system-level configuration changes, launching the interpreter is once again causing crashes. In addition to the crash log I pasted earlier, I see this in the console when launching the interpreter:
$ python3.6
Fatal Python error: Py_Initialize: unable to load the file system codec
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/__init__.py", line 33, in <module>
ImportError: cannot import name 'aliases'
zsh: abort python3.6

Indeed it seems that aliases.py is missing.
$ ls /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/a*
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/ascii.py

Hmm. Looks like I may [have a trojan](http://stackoverflow.com/a/4707258).

Jason, assuming you are using the python.org 3.6.0r1 installer, let us know what you find out!

I am using the python.org rc1 installer, confirmed to match the public md5 sum:
$ md5 ~/Downloads/python-3.6.0rc1-macosx10.6.pkg 
MD5 (/Users/jaraco/Downloads/python-3.6.0rc1-macosx10.6.pkg) = 404c390ae27f067aaab34f168cf913eb
I downloaded ClamXav and had it scan my entire system. It found nothing of interest, only two Windows exploits, one in a driver from Asus and another in an attachment to an e-mail that was already filed as junk. It also found two "Eicar-Test-Signature" hits in old IRC logs.
So if my Mac has a virus, it's new enough that it's not being detected by ClamXav.
I'm going to continue to investigate and see if I can detect what event triggers the removal of that file.

I thought I was on to something when I found this technique for using dtrace to detect a file deletion: https://blogs.oracle.com/zoneszone/entry/who_keeps_removing_that_file
However, I don't seem to have privilege to run it.
$ cat trap-aliases-delete 
#! /usr/sbin/dtrace -wqs
syscall::unlinkat:entry
/cleanpath(copyinstr(arg1)) == "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/aliases.py"/
{
 stop();
 printf("%s[%d] caught removing aliases.py\n", execname, pid);
 system("ptree %d; pstack %d; prun %d", pid, pid, pid);
}
$ sudo ./trap-aliases-delete
dtrace: system integrity protection is on, some features will not be available
dtrace: could not enable tracing: Permission denied

Looks like I can [disable SIP](http://internals.exposed/blog/dtrace-vs-sip.html) and dtrace will be viable.

Before disabling SIP, I found I could use fs_usage to get some visibility into fs changes:
sudo fs_usage -w -f filesys | grep aliases.py
To test my understanding of fs_usage, I deleted aliases.py (also confirming that triggers the error). When I did so, I noticed something interesting - my Dropbox account detected deletion of 34 instances of a file called aliases.py. A quick search reveals that I have about 34 virtualenvs created by tox, each of which links .tox/python/lib/python3.6/encodings to /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings.
Now I'm suspecting that there's a race condition where Dropbox ends up deleting the system file. Either that, or there's a buggy piece of code somewhere that's deleting encodings/aliases.py (and possibly other files) in one of these virtualenvs, causing the file to be deleted in the system.
I'm using Dropbox v16.3.27 (https://www.dropboxforum.com/t5/Desktop-client-builds/Beta-Build-16-3-27/td-p/197057).
I'll continue to investigate, but I'm becoming increasingly suspicious of this being an environmental problem.

I've just encountered what appears to be a different manifestation of the same issue. I unlinked the .tox directory and that rendered importlib broken.
$ pwd
/Users/jaraco/yg/queso
$ rm -R .tox
$ python
Python 3.6.0rc1 (v3.6.0rc1:29a273eee9a5, Dec 6 2016, 16:24:13) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ python -m tox
Could not import runpy module
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 15, in <module>
 import importlib.util
ModuleNotFoundError: No module named 'importlib.util'
$ ls /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib
__init__.py		__pycache__		_bootstrap.py		_bootstrap_external.py	abc.py			machinery.py
Although I'm not certain that another process wasn't involved, I'm confident that it was the removal of the .tox directory (and interaction with Dropbox) that caused the corruption.

That's sounding better, thanks. Out of curiosity, are there hard links involved?

I don't think hard links are involved. I see symlinks in the .tox/python virutalenv, and virtualenv defaults to creating symlinks (https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L565-L570) and falls back to copying files rather than hard linking.

I've reinstalled Dropbox, downgrading to 15.4.22. I'll see if that helps.

Downgrading Dropbox did not help. This morning I woke up to `importlib.abc` missing.

I'm getting good at reinstalling Python, though. Now I'm invoking it with this single command-line:
sudo installer -verboseR -pkg ~/Downloads/python-3.6.0rc1-macosx10.6.pkg -target /

Have you verified that the problem no longer occurs if the virtualenvs are not in a Dropbox directory? (And these are virtualenvs, not venvs?) In any case, at this point it seems pretty clear that we're not dealing with a problem with Python 3.6.0 nor with the python.org installer, no?

This morning, I noticed my Mercurial install isn't working either - failing with this error message:
$ hg pull
abort: couldn't find mercurial libraries in [/usr/local/Cellar/mercurial/4.0.1/lib/python2.7/site-packages /usr/local/Cellar/mercurial/4.0.1/bin /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python27.zip /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7 /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload /Users/jaraco/Library/Python/2.7/lib/python/site-packages /usr/local/lib/python2.7/site-packages /Library/Python/2.7/site-packages]
(check your install and PYTHONPATH)
That happens even though I was still able to "import mercurial" in my Python environment. I uninstalled and reinstalled the Mercurial (using Homebrew) and it failed to reinstall with the same error, suggesting strongly that it was the Python environment that was corrupted. I then uninstalled and reinstalled the Homebrew Python 2.7, and the problem was solved. I think this points to another manifestation of something deleting files that were symlinked by directories in the virtualenv.

> virtualenvs and not venvs? 
Correct.
> In any case, at this point it seems pretty clear that we're not dealing with a problem with Python 3.6.0 nor with the python.org installer, no?
Also correct.
If you don't mind, I'd like to continue to document and log this issue here. I've removed Python 3.6 as implicated and I'll remove you as nosy.

Thanks. I'm still interested in the outcome but my immediate concern was whether this was a showstopper problem for 3.6.0. Good luck!

I paused the Dropbox sync and ran the `rm -R .tox; tox` routine many times without incident, which strongly implicates Dropbox. I'm going to now re-enable Dropbox sync and see if I can trigger the behavior.
Now even with Dropbox updated back to the latest version (16.3.27) and performing the same operations I was two days ago, I've been unable to elicit the behavior. I've confirmed I still have access to delete those files that were being deleted, but yet I can't trigger the failure. Perhaps whatever environmental thing that happened on Monday was a fluke and I'll go back not to being bothered ever by this condition.

Throughout the week, I've not experienced this issue again. Perhaps the reinstall of Dropbox corrected some transient condition. Or perhaps my system was in some environment where it was particularly sensitive on Monday but has since returned to the status quo. In any case, I'm going to close this issue until such a time that it occurs again.