-
-
Notifications
You must be signed in to change notification settings - Fork 8k
Port build system to Meson #26621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Port build system to Meson #26621
Conversation
And looking back at #23829, I forgot about adding the dev extra.
1ba714f to
ee25941
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have an if __name__ == '__main__': guard
ee25941 to
5b94c3b
Compare
86b2d7b to
7f5d3a9
Compare
9eaf930 to
cb71290
Compare
cb71290 to
1fa418e
Compare
OK, I think we're at a good point now. A couple of issues had to be cleared up:
- I accidentally forgot to install the
macosxbackend, which made it unavailable in tests, so auto-picking in subprocesses chose TkAgg, which is still broken on Azure. I have fixed the backend here, but also made the subprocess explicitly use Agg where possible in Fix flaky CI tests #26680 . Unfortunately, I don't think we can fixtest_lazy_auto_backend_selectionas its point is to not have a backend set. - In some CI, we use the backend override from
mplsetup.cfg; I've replicated that using the Meson equivalent, but I wonder if we should do that everywhere. - Azure and AppVeyor have a ton of compilers preinstalled in addition to MSVC. Meson picks the non-MSVC option by default, so I have explicitly passed the option to use MSVC in CI. Since we don't specifically hard-require MSVC, I have not set that option in our config, and I expect most developers do not have multiple compilers on Windows, but it is a gotcha on CI.
There are also a couple followups that need to be considered eventually:
- Versions are not dynamically generated at build time. At runtime, we use
setuptools-scm, so it is fine for editable installs. However, this does mean that wheels all use3.9.0.dev0as version, which reduces utility of the nightly wheels slightly. I'm not sure if we can hack around this post-build. Upstream is working on a better API for this: Support dynamic version generation, including from git tags and with git commit hashes mesonbuild/meson-python#159 - I don't think all license files are installed the same way as setuptools. I'm not sure if we have any easy way to fix that right now. There is an uptsream issue for this as well: Allow
licenseto be listed in the dynamic section mesonbuild/meson-python#270
There isn't a huge change for developers, and you might be able to glean that from the CI changes (though most changes are extraneous for local work.) There may be a bit more change for distributions, but they might be used to that from NumPy/SciPy's existing change. I will write up some docs for that next week.
1fa418e to
29c2b60
Compare
OK, there is actually one stall on the macOS arm64 builds, which I've reported upstream at mesonbuild/meson-python#468
I have found a solution that appears to work regarding getting stubtest running with editable install, though not sure its actually a good solution...
If you add the $REPO_ROOT/lib to the mesonpy-generated matplotlib-editable.pth file in site-packages, stubtest is able to run and give accurate results. (One slight change from setuptools editable install is that we had unused allowlist entries for untyped subpackages, but that is pretty trivial)
As far as I can tell it does not affect the special handling set up by mesonpy in _matplotlib_editable_loader, but does allow mypy to connect the stubs to the runtime.
All that said, while it may be good enough for a controlled CI environment, I don't like that as a solution for developers who may wish to run stubtest on their machine (note that mypy alone will work fine, just not with -p or -m, you must give it the lib/matplotlib path... stubtest requires runtime, so is more restricted) In my testing my editor integration still worked fine without any changes.
Perhaps this is a safe enough change and we should lobby/PR upstream to include it to support the mypy usecase.
The other option which seems to also be broken right now, but I don't think should be a problem, is to do a non-editable install... this is not great for the developer machine use case either, but at least does not involve editing generated files...
Currently I'm getting:
error: matplotlib failed to import, InvalidVersion: Invalid version: '"3.9.0.dev0"'
Which is a weird error on a couple fronts, firstly being why is version needed for importing in the first place?, secondly being why is that an invalid version?
Investigation has revealed that stubtest is doing (the __import__ equivalent of) from matplotlib import *, and that is what is triggering the "InvalidVersion". Testing on a REPL confirms that while import matplotlib is fine, trying to get __version_info__ (including a * import) breaks. The problem appears to be that there are quotes on __version__.
In summary:
- I think after sorting out
__version_info__problems probably the best short term path for CI is to use non-editable install for the stubtest job - If that doesn't work, editing the
pthfile should - May be worth talking to upstream about this usecase and seeing if the addition to the pth file can be included there, which would help for developer machines without having to edit a generated file.
89d9351 to
e50c4dd
Compare
Cherry-picked out of matplotlib#26800; also unpin setuptools.
e50c4dd to
01c1ff6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be OK now? To some extent it may be better to try and get this in so that there is ample time before the 3.9-release to find any issues, but I am still a bit worried to merge it myself...
Well, what is the worst that can happen?
To some extent it may be better to try and get this in so that there is ample time before the 3.9-release to find any issues, but I am still a bit worried to merge it myself...
Well, what is the worst that can happen?
I would not be worried. You had another approving review and merged it as the maintainer's workflow says. As you say, this will give a good opportunity to see if anything does crop up rather than waiting until the last minute.
This removes special casing for AIX and IBM i OS, but the FreeType build is using an included Meson system instead of their own old autotools-based build.
@ayappanec @zheddie can you confirm whether the main branch still builds for you?
@KamathForAIX Can you confirm whether the main branch builds fine in AIX ?
Hi @QuLogic ,
So in AIX gcc with LTO option is not supported. So in meson.build file we should not use it as a default option if the target is AIX. If we can write a logic like that it would be great.
Secondly, in the matplotlib/tools/generate_matplotlibrc.py I had to change !/usr/bin/env python to /usr/bin/env python3.9..
With the two changes mentioned above we are able to build in AIX. Kindly let me know what we can do regarding the same.
Does !/usr/bin/env python3 work (rather than python3.9)?
@tacaswell Yes it works
So in AIX gcc with LTO option is not supported. So in meson.build file we should not use it as a default option if the target is AIX. If we can write a logic like that it would be great.
If that is not supported, then this should be handled upstream in Meson. If they silently ignore b_lto=true, then we will have to do nothing. If they error out, then we should see if we can conditionalize it.
Hi @QuLogic ,
Sure. We will work this out with the meson community members. No worries. Thank you for connecting with AIX for the meson build system test for Matplotlib. If there is anything update or help needed we will get back to you.
Is there anything else we can do for Matplotlib community from AIX? Kindly let us know.
Have a nice day ahead.
Regards,
Aditya Kamath.
@KamathForAIX I do not know of any CI service that offer AIX workers so we are a bit blind to if we accidentally break anything. If you could have a monthly cron-job that tries to build Matplotlib and run the tests and report any failures up to us that would be super helpful!
Hi @tacaswell
Sure, we will set a CI for matplotlib and run the tests.
Uh oh!
There was an error while loading. Please reload this page.
PR summary
Now that NumPy and SciPy have spent some time on Meson, it's not too bad to move our build system over to Meson as well. This PR also implements the move to
pyproject.tomlfrom #23829. I have added some documentation with some key points for migration, though most are edge cases that most developers shouldn't have to worry about.The main change is that editable installs should disable build isolation:
This is because
pyproject.toml-based install would normally isolate builds and extensions are built in the isolated environment, which goes against the idea of an editable install. (You will also need to install build requirements in your target environment, but I think that's not new anyway.)The other downside is that all files need to be listed manually in most
meson.build. However, this also has a side benefit of not accidentally installing extra files (e.g., I get several temporary*.gcnofiles and the wholenode_modulesdirectory if they happen to exist onmain.)The upside is that compilation is written in a very standard way. Both FreeType and Qhull are subprojects using manually written
meson.buildbut those are just the same as the files in the main source. Qhull's build is a simple port of what we did insetupext.pytomeson.build. FreeType's build is based on the 2.9.1 version in Meson's wrapdb, but dropping some of the extra install bits we won't use and rolled back to 2.6.1. For 2.11.1, we use the Meson build system already part of FreeType. When we move forward on FreeType versions, we can drop the patched-in build and use the upstream build system.The other upside of Meson+meson-python builds is that compilation is a) faster with better parallelism since the entire set of files can be seen by the build system, and b) can be done at import time in an editable install. The latter means that you will no longer need to re-install when editing C(++) files (unless you switch to an older commit.)
Closes #23829
PR checklist