homepage

I am using OSX 10.10.5 (14F27) and Python 3.5. Python 3.5 was installed via brew.
When greenlet is installed inside virtualenv using py3.5 header files are
missing inside /path/to/venv/include/python3.5m. As a result uwsgi is unable to build with asyncio support
~ brew info python3
python3: stable 3.5.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python3/3.4.3 (4721 files, 82M)
Poured from bottle
/usr/local/Cellar/python3/3.4.3_2 (5948 files, 98M)
Built from source
/usr/local/Cellar/python3/3.5.0 (3573 files, 61M) *
Poured from bottle
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python3.rb
==> Dependencies
Build: xz , pkg-config 
Required: openssl 
Recommended: readline , sqlite , gdbm , xz 
Optional: homebrew/dupes/tcl-tk
Steps to reproduce
------------------
1. Create a venv
$/tmp python3.5 -m virtualenv venv_bug
2. Install greenlet
$/tmp . venv_bug/bin/activate
(venv_bug)$/tmp pip install greenlet
Collecting greenlet
 Using cached greenlet-0.4.9.tar.gz
Building wheels for collected packages: greenlet
 Running setup.py bdist_wheel for greenlet
 Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-sxtdlufe/greenlet/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpf6g8rwt_pip-wheel-:
 running bdist_wheel
 running build
 running build_ext
 building 'greenlet' extension
 creating build
 creating build/temp.macosx-10.10-x86_64-3.5
 clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c greenlet.c -o build/temp.macosx-10.10-x86_64-3.5/greenlet.o
 creating build/lib.macosx-10.10-x86_64-3.5
 clang -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.5/greenlet.o -o build/lib.macosx-10.10-x86_64-3.5/greenlet.cpython-35m-darwin.so
 installing to build/bdist.macosx-10.10-x86_64/wheel
 running install
 running install_lib
 creating build/bdist.macosx-10.10-x86_64
 creating build/bdist.macosx-10.10-x86_64/wheel
 copying build/lib.macosx-10.10-x86_64-3.5/greenlet.cpython-35m-darwin.so -> build/bdist.macosx-10.10-x86_64/wheel
 running install_headers
 creating build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data
 creating build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data/headers
 copying greenlet.h -> build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data/headers
 running install_egg_info
 running egg_info
 creating greenlet.egg-info
 writing top-level names to greenlet.egg-info/top_level.txt
 writing greenlet.egg-info/PKG-INFO
 writing dependency_links to greenlet.egg-info/dependency_links.txt
 writing manifest file 'greenlet.egg-info/SOURCES.txt'
 warning: manifest_maker: standard file '-c' not found
 reading manifest file 'greenlet.egg-info/SOURCES.txt'
 reading manifest template 'MANIFEST.in'
 writing manifest file 'greenlet.egg-info/SOURCES.txt'
 Copying greenlet.egg-info to build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9-py3.5.egg-info
 running install_scripts
 Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-sxtdlufe/greenlet/setup.py", line 101, in <module>
 **setuptools_args)
 File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
 dist.run_commands()
 File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
 self.run_command(cmd)
 File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
 cmd_obj.run()
 File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 213, in run
 archive_basename = self.get_archive_basename()
 File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 161, in get_archive_basename
 impl_tag, abi_tag, plat_tag = self.get_tag()
 File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
 assert tag == supported_tags[0]
 AssertionError
 ----------------------------------------
 Failed building wheel for greenlet
Failed to build greenlet
Installing collected packages: greenlet
 Running setup.py install for greenlet
Successfully installed greenlet-0.4.9
3. Install uWSGI
(venv_bug)$ /tmp CFLAGS="-I/tmp/venv_bug/include/python3.5m" UWSGI_PROFILE="asyncio" pip install uwsgi
Collecting uwsgi
 Using cached uwsgi-2.0.11.2.tar.gz
Building wheels for collected packages: uwsgi
 Running setup.py bdist_wheel for uwsgi
 Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpwoq08qywpip-wheel-:
 running bdist_wheel
 running build
 running build_py
 creating build
 creating build/lib
 copying uwsgidecorators.py -> build/lib
 installing to build/bdist.macosx-10.10-x86_64/wheel
 running install
 plugins/greenlet/greenlet.c:2:10: fatal error: 'greenlet/greenlet.h' file not found
 #include <greenlet/greenlet.h>
 ^
 1 error generated.
 ----------------------------------------
 Failed building wheel for uwsgi
Failed to build uwsgi
Installing collected packages: uwsgi
 Running setup.py install for uwsgi
 Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools, tokenize;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-69gibzr6-record/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/venv_bug/bin/../include/site/python3.5/uwsgi:
 running install
 plugins/greenlet/greenlet.c:2:10: fatal error: 'greenlet/greenlet.h' file not found
 #include <greenlet/greenlet.h>
 ^
 1 error generated.
 ----------------------------------------
Command "/private/tmp/venv_bug/bin/python3.5 -c "import setuptools, tokenize;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-69gibzr6-record/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/venv_bug/bin/../include/site/python3.5/uwsgi" failed with error code 1 in /private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi
Also setuptools is at latest version 18.2.
Installing uwsgi site wide succeeds. greenlet directory is present inside `python3.5m`.
(venv_bug)$ /tmp ls -la /usr/local/include/python3.5m | grep greenlet
drwxr-xr-x 3 root admin 102 Nov 1 20:40 greenlet
Links:
uWSGI asyncio: https://uwsgi-docs.readthedocs.org/en/latest/asyncio.html#building-uwsgi-with-asyncio-support
Greenlet issue thread: https://github.com/python-greenlet/greenlet/issues/96 

Just wanted to clarify that greenlet.h is not missing in virtualenv, but headers seem to be installed in venv/include/site/python3.5, when only venv/include is considered for includes when building extensions. It is not specific to OSX and it is trivial to reproduce on Linux too.

Greenlet isn't part of python, so what bug in CPython are you reporting? It sounds like you should be reporting this to the greenlet folks.

I have attached the link to greenlet issue as well. The issue seems to be with distutils/setuptools.

I don't see a link to a greenlet issue, and setuptools is not maintained as part of the stdlib. It sounds like we should close this until and if the greenlet project can point us to a specific problem with the stdlib distutils.

Link to greenlet issue: https://github.com/python-greenlet/greenlet/issues/96 

I have created an issue in setuptools repo: https://bitbucket.org/pypa/setuptools/issues/455/unable-to-build-extension-in-python-35.

Thanks for the links. I'll leave this open for now since it isn't clear where the problem is. Please update the issue with any additional information you get from setuptools if it looks like a distutils problem, or ping this issue if you get no response there for an extended period.

As David notes, this issue does not document any problems with Python 3.5 itself. I see at least two issues here. One, you are using virtualenv, a third-party package, rather than Python's built-in venv. It appears that the most recent release of virtualenv on PyPI, virtualenv 13.1.2, includes outdated wheels of other third-party products, in particular, of wheel: 0.24, instead of the current 0.26 (https://pypi.python.org/pypi/wheel). If you update wheel in your virtualenv that should fix the "assert tag == supported_tags[0]" issue. Suggest you open an issue against virtualenv to ask that they update their vendored packages or perhaps not include them. Another problem, when using the python.org python3.5, is with the install script of the uwsgi, uwsgiconfig.py. The script does some machinations to figure out what compiler it is using. Unfortunately, it seems to get confused by the fact that, with the last several major releases of Xcode, Apple no long ships gcc but does provide a gcc alias that actually invokes clang. If you override the CC value and explictly set CC=clang, the uwsgi install seems to work correctly with the python.org OS X 3.5.0. You should probably open an issue with the uwsgi project about that. I do not have a homebrew installation handy to check, so if there are other problems after updating wheel in your virtualenv (or switching to use builtin venv) and using CC=clang, suggest you ask on a homebrew forum.

I have reproduced this problem on Linux with pyvenv, isn't that part of Python 3.5?

I created a simple gist that can show the problem: https://gist.github.com/snaury/ea167713b1804ffbaf5a (testme.sh builds Python 3.5, creates a venv, install greenlet and tries to compile an extension)

Alexey, thanks for the simplified test case. I'll look at it further.

Updating wheel to latest version fixed assertion issue but uwsgi installation still fails.
Virtualenv has updated to latest whhel version https://github.com/pypa/virtualenv/blob/develop/virtualenv_support/wheel-0.26.0-py2.py3-none-any.whl. Also, my example used `python -m venv`, this is part of stdlib and using `pyvenv-3.5` produced same result.
Explicitly setting `CC=clang` in OSX didn't help https://gist.github.com/kracekumar/270533354d62be46f77d and also on Ubuntu 14.04 produces same error with gcc and clang.
I have opened an issue in uwsgi https://github.com/unbit/uwsgi/issues/1100.

I've looked at this some more using variations of Alexy's test case and I now think there are at least two problems here. And both issues have to do with confusion about exactly where a distribution's header files should be installed in venvs (created with the standard library venv) or virtual environments (created with the third-party virtualenv).
One, when building an extension module from a venv, py3 Distutils tries to add the path of the venv include directory to the list of include directories supplied to the C compiler front-end. However, build_ext assumes that the include files have been installed in venv/include (Lib/distutils/command/build_ext.py#l157) while install.py uses an "install scheme"-specific location to install headers (Lib/distutils/command/install.py#l22); for unix non-user installs, that may be something like venv/include/python3.5m. This is particularly a problem for a situation like here where the build of an extension module from one distribution, uwsgi, is dependent on the header files for an extension module from a previously installed distribution, greenlet. So, to resolve this, it seems like build_ext needs to be smarter about the include path for venvs. There should also be test case(s) to ensure that install_headers and build_ext are using the same paths in all relevant environments, e.g. venv, non-venv, user install, Windows, etc.
But even if that is fixed, there is another issue. It appears pip uses yet another location for installing distribution header files in virtualenvs and venvs: venv/include/site/python3.5. So, if pip is used to install the distribution supplying the include files, the dependent distribution will still fail to build. From the deleted comments in a recent commit (https://github.com/pypa/pip/commit/882cd358d1abd5e42df6333dddc42f52ab7d6ff2), it appears the reason why pip does this is that virtualenv creates a symlink to the interpreter's global include directory which means that include files from distributions installing to the virtualenv cannot be written to the normal (non-virtualenv) path. It also looks like venv does not create such a symlink but depends on Distutils build_ext to supply paths to both the venv include directory and the global include directory to compilers. I'm not sure what the right solution is here and I certainly may be missing something. In any case, it must be possible for distributions to find the header files from other distributions regardless how they were installed.
Since these are all packaging issues, I think the PyPA folks need to take this and decide what action(s) are needed where.

Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.
If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools