-
-
Notifications
You must be signed in to change notification settings - Fork 58
Automatic fix and detection of partial installations, option to bundle extra wheels for simpler offline and monorepo builds #169
-
Edit (14/04/2025): Apologies for the rambled write-up. In hindsight, not really something that fits the direction of the project. I've since moved on from PyApp, feel free to delete this discussion. Appreciate your time and work regardless!
Original comment (click to expand)
Hi, I've got a fork with two interesting features that adds extra safety and flexibility for monorepos and offline installations.
Pardon the length, I like to add reasoning and context!
Partial installation detection
Problem: Whenever pyapp is closed mid-way an installation, the heuristic in ensure_ready() of checking for the existence of !app::install_dir().is_dir() fails to notice this partial installation, and subsequent runs gives module not found (as it's not.. installed). The only way out is doing a self restore or manually deleting the installation path.
While this is trivial for us programmers, giving such instructions to (Windows) users are annoying and error-prone
Solution: A ready.flag file is created in the root of the installation after install_project(), and ensure_ready() logic now checks for its existence before materialize() and creates it after so. The file works like a checkpoint ensuring previous runs reached the end of the installation correctly! :)
I'm not sure if this is the best place to put it, maybe calling self restore at the beginning of run_project() which also works, but could break isolation compatibility
In the same idea, we can save the current binary hash to a file and call self restore if a same-version but different contents release was run (mostly in dev mode), I also have coded it, but within Python this time, and it works perfectly.
Bundling extra wheels
Problem: In a monorepo scenario like mine, I'm having to bundle all sub-projects packages into a monolithic wheel in PYAPP_PROJECT_PATH with [tool.hatch.build.targets.wheel], and defining main-lib[all] dependency to be installed only for executable releases (dynamically editing pyproject.toml pre-build), this is clunky to say the least.
Solution: While the previous works, having an option to bundle multiple local wheels like in PYAPP_PROJECT_PATH="/A.whl;/B.whl;C.whl.." would solve this issue, just list all wheels out of uv build --wheel --all and do a single pip install *.whl in pyapp when bootstrapping the pyapp project.
The implementation I made is slightly different, I added a
PYAPP_EXTRA_WHEELS=a;b;cenv that gets installed afterPYAPP_PROJECT_PATH, mostly due the already implement complexity and assumptions of the later. But I think the current logic could grab the first file of this new proposed semicolon separatedPYAPP_PROJECT_PATHas the main file?
The effects and possibilities of this are:
- If we bundle all packages of
uv tree, we pretty much created an easier method for making offline distributions - Simplifies the build process for monorepos substantially, which
uvupstreams and promotes it a lot:- It now finds all workspace members than listing manually, allowing for private projects as well
- No need to list
main-lib[all], as the subprojects defines optional deps of the bundledmain-lib
The important thing is to pip install all bundled wheels in a single go, as individual wheels would look for pypi for deps not already installed! So joining EXTRA_WHEELS and PROJECT_PATH could do it, we don't need to modify PROJECT_PATH, keeping current compatibility of how it works in essence.
Thanks a lot for the work and attention, I hope you consider adding these mechanisms as they'd help me a lot ✨
You can base-off the work in my proof of concept no problem if you do it!
Beta Was this translation helpful? Give feedback.