-
Notifications
You must be signed in to change notification settings - Fork 1.1k
mip: Support relative source file paths in package.json. #676
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mip: Support relative source file paths in package.json. #676
Conversation
This adds the ability for `mip` to load files from _relative_ source paths given in the `urls` section of `package.json` files. Previously, the only accepted forms for source paths were "http://", "https://", or "github:". These changes allow a source path to be given without one of these prefixes, in which case the URL of the parent of the `package.json` is prepended. So instead of someone forking a repository and having to change source lines that start with github:upstream_repository/filename into github:my_repository/filename they can now just say filename which will survive forking or cloning without change. This also allows doing `mip.install()` from a `file://` location. This makes it easy to do a `git clone` of a repository and test installation of a package from the local filesystem, especially using the unix port. Signed-off-by: Ned Konz <ned@productcreationstudio.com>
Adding support for installation from a local folder has been on my to-do list, thanks for adding it here.
I'm also keen to support direct install from manifest rather than needing the json file but it feels right to add just the relative & local path stuff as a clean change first. I'll still try to look into inline manifest support myself on top of this eventually!
Actually, directly creating a package.json from a manifest.py doesn't really cover the entire requirement.
As a package developer, I want to ignore .mpy files and other intermediate/implementation details, concentrating on the source files. However, the source files may well include:
- Python files, which get handled pretty well presently, both in
micropython-libdistribution and in the micropython build process - Binary resources, which need to be converted to Python using
mpy_bin2res.pyand loaded usingpkg_resources - Other files that need special processing (like the conversion of
utemplate.tplfiles into_tpl.pygenerators usingutemplate_util.py
The problem is that the manifest.py doesn't allow for these other conversion steps (some of which may need to be specified in a package-dependent manner, depending on the nature of the source files).
And whatever scheme is developed will have to deal both with building into frozen flash as well as distribution via files or networking.
I guess this is something that needs a broader discussion with the community.
I know support for freezing non-python resources is included in this PR: micropython/micropython#8381
This will presumably include the manifest support for describing said resources, however I don't know if conversion across to json or mip handling has been considered.
The trick of converting/wrapping them in a python file is a neat workaround for now, but ultimately inefficient and fiddly.
That being said it can be quite handy, I've actually just used the same binary 2 python concept to wrap up libraries and files to build a cpython app as a single static exe on desktop with pyoxidizer :-)
OK, I tested some additional cases and found that I was getting OSError -113 (and/or 118?) from time to time.
I added retries around these errors after a gc.collect() and a sleep, which seems to have increased reliability on my ESP32.
I would like to add an optional third field for the contents of urls that would allow for secondary github: URLs (those not from the same repo as the package.json) to specify a non-HEAD branch. My current PR just sets the branch for secondary github: URLs to HEAD.
This PR adds the ability for
mipto load files from relative source paths given in theurlssection ofpackage.jsonfiles.Previously, the only accepted forms for source paths were
http://,https://, orgithub:.These changes allow a source path to be given without one of these prefixes, in which case the URL of the parent of the
package.jsonis prepended.So instead of someone forking a repository and having to change source lines that start with
github:upstream_repository/filenameinto
github:my_repository/filenamethey can now just say
filenamewhich will survive forking or cloning without change.
This also allows doing
mip.install()from afile://location. This makes it easy to do agit cloneof a repository and test installation of a package from the local filesystem, especially using the unix port.To install this package from my fork for testing, (I think you can) use commands such as:
Or from a networked device:
And then restart your device, as you have now added a new version of
mip. To test this, you will need to make sure that yourlibdirectory comes before.frozenin yoursys.pathbefore importingmip.