[Python-checkins] peps: Account for metadata hook deferral in PEP 426

Sun Mar 2 07:46:40 CET 2014

http://hg.python.org/peps/rev/272debef6b77
changeset: 5394:272debef6b77
user: Nick Coghlan <ncoghlan at gmail.com>
date: Sun Mar 02 16:46:28 2014 +1000
summary:
 Account for metadata hook deferral in PEP 426
files:
 pep-0426.txt | 428 ++++++++++++++++++++------------------
 1 files changed, 225 insertions(+), 203 deletions(-)

diff --git a/pep-0426.txt b/pep-0426.txt
--- a/pep-0426.txt
+++ b/pep-0426.txt
@@ -1200,6 +1200,12 @@
 ``extensions`` key. The keys MUST be valid prefixed names, while
 the values MUST themselves be nested mappings.
 
+Two key names are reserved and MUST NOT be used by extensions, except as
+described below:
+
+* ``extension_version``
+* ``required_extension``
+
 The following example shows the ``python.details`` and ``python.commands``
 standard extensions from :pep:`459`::
 
@@ -1235,6 +1241,14 @@
 the meaning of the extension. This practice will also make it easier to
 find authoritative documentation for metadata extensions.
 
+Metadata extensions allow development tools to record information in the
+metadata that may be useful during later phases of distribution, but is
+not essential for dependency resolution or building the software.
+
+
+Extension versioning
+--------------------
+
 Extensions MUST be versioned, using the ``extension_version`` key.
 However, if this key is omitted, then the implied version is ``1.0``.
 
@@ -1248,9 +1262,24 @@
 extension metadata using the lowest metadata version that includes
 all of the needed fields.
 
-Metadata extensions allow development tools to record information in the
-metadata that may be useful during later phases of distribution, but is
-not essential for dependency resolution or building the software.
+
+Required extensions
+-------------------
+
+A project may consider correct handling of some extensions to be essential
+to correct installation of the software. This is indicated by setting the
+``required_extension`` field to ``true``. Setting it to ``false`` or
+omitting it altogether indicates that processing the extension when
+installing the distribution is not considered mandatory by the developers.
+
+Installation tools MUST fail if ``required_extension`` is set to ``true``
+for an extension and the tool does not have any ability to process that
+particular extension (whether directly or through a tool-specific plugin
+system).
+
+If an installation tool encounters a required extension it doesn't
+understand when attempting to install from a wheel archive, it MAY fall
+back on attempting to install from source rather than failing entirely.
 
 
 Extras (optional dependencies)
@@ -1569,8 +1598,6 @@
 
 * Updated obsolescence mechanism
 
-* Metadata hook system
-
 * More flexible system for defining contact points and contributors
 
 * Defined a recommended set of project URLs
@@ -1709,20 +1736,199 @@
 particular extensions to be provided as optional features.
 
 Possible future uses for extensions include declaration of plugins for
-other distributions, hints for automatic conversion to Linux system
-packages, and inclusion of CVE references to mark security releases.
+other distributions and hints for automatic conversion to Linux system
+packages.
+
+The ability to declare an extension as required is included primarily to
+allow the definition of the metadata hooks extension to be deferred until
+some time after the initial adoption of the metadata 2.0 specification. If
+a distribution needs a ``postinstall`` hook to run in order to complete
+the installation successfully, then earlier versions of tools should fall
+back to installing from source rather than installing from a wheel file and
+then failing to run the expected postinstall hook.
+
+
+Changes to environment markers
+------------------------------
+
+There are three substantive changes to environment markers in this version:
+
+* ``platform_release`` was added, as it provides more useful information
+ than ``platform_version`` on at least Linux and Mac OS X (specifically,
+ it provides details of the running kernel version)
+* ordered comparison of strings is allowed, as this is more useful for
+ setting minimum and maximum versions where conditional dependencies
+ are needed or where a platform is supported
+* comparison chaining is explicitly allowed, as this becomes useful in the
+ presence of ordered comparisons
+
+The other changes to environment markers are just clarifications and
+simplifications to make them easier to use.
+
+The arbitrariness of the choice of ``.`` and ``_`` in the different
+variables was addressed by standardising on ``_`` (as these are all
+predefined variables rather than live references into the Python module
+namespace)
+
+The use of parentheses for grouping was explicitly noted to address some
+underspecified behaviour in the previous version of the specification.
+
+
+Updated contact information
+---------------------------
+
+This feature is provided by the ``python.project`` and
+``python.integrator`` extensions in :pep:`459`.
+
+The switch to JSON made it possible to provide a more flexible
+system for defining multiple contact points for a project, as well as
+listing other contributors.
+
+The ``type`` concept allows for preservation of the distinction between
+the original author of a project, and a lead maintainer that takes over
+at a later date.
+
+
+Changes to project URLs
+-----------------------
+
+This feature is provided by the ``python.project`` and
+``python.integrator`` extensions in :pep:`459`.
+
+In addition to allow arbitrary strings as project URL labels, the new
+metadata standard also defines a recommend set of four URL labels for
+a distribution's home page, documentation, source control and issue tracker.
+
+
+Changes to platform support
+---------------------------
+
+The new environment marker system makes it possible to define supported
+platforms in a way that is actually amenable to automated processing. This
+has been used to replace several older fields with poorly defined semantics.
+
+For the moment, the old ``Requires-External`` field has been removed
+entirely. The metadata extension mechanism will hopefully prove to be a more
+useful replacement.
+
+
+Updated obsolescence mechanism
+------------------------------
+
+The marker to indicate when a project is obsolete and should be replaced
+has been moved to the obsolete project (the new ``obsoleted_by`` field),
+replacing the previous marker on the replacement project (the removed
+``Obsoletes-Dist`` field).
+
+This should allow distribution tools to more easily warn users of
+obsolete projects and their suggested replacements.
+
+The ``Obsoletes-Dist`` header is removed rather than deprecated as it
+is not widely supported, and so removing it does not present any significant
+barrier to tools and projects adopting the new metadata format.
+
+
+Included text documents
+-----------------------
+
+This feature is provided by the ``python.details`` extension in :pep:`459`.
+
+Currently, PyPI attempts to determine the description's markup format by
+rendering it as reStructuredText, and if that fails, treating it as plain
+text.
+
+Furthermore, many projects simply read their long description in from an
+existing README file in ``setup.py``. The popularity of this practice is
+only expected to increase, as many online version control systems
+(including both GitHub and BitBucket) automatically display such files
+on the landing page for the project.
+
+Standardising on the inclusion of the long description as a separate
+file in the ``dist-info`` directory allows this to be simplified:
+
+* An existing file can just be copied into the ``dist-info`` directory as
+ part of creating the sdist
+* The expected markup format can be determined by inspecting the file
+ extension of the specified path
+
+Allowing the intended format to be stated explicitly in the path allows
+the format guessing to be removed and more informative error reports to be
+provided to users when a rendering error occurs.
+
+This is especially helpful since PyPI applies additional restrictions to
+the rendering process for security reasons, thus a description that renders
+correctly on a developer's system may still fail to render on the server.
+
+The document naming system used to achieve this then makes it relatively
+straightforward to allow declaration of alternative markup formats like
+HTML, Markdown and AsciiDoc through the use of appropriate file
+extensions, as well as to define similar included documents for the
+project's license and changelog.
+
+Grouping the included document names into a single top level field gives
+automated tools the option of treating them as arbitrary documents without
+worrying about their contents.
+
+Requiring that the included documents be added to the ``dist-info`` metadata
+directory means that the complete metadata for the distribution can be
+extracted from an sdist or binary archive simply by extracting that
+directory, without needing to check for references to other files in the
+sdist.
+
+
+Appendix D: Deferred features
+=============================
+
+Several potentially useful features have been deliberately deferred in
+order to better prioritise our efforts in migrating to the new metadata
+standard. These all reflect information that may be nice to have in the
+new metadata, but which can be readily added in metadata 2.1 without
+breaking any use cases already supported by metadata 2.0.
+
+Once the ``pypi``, ``setuptools``, ``pip``, ``wheel`` and ``distlib``
+projects support creation and consumption of metadata 2.0, then we may
+revisit the creation of metadata 2.1 with some or all of these additional
+features.
+
+
+MIME type registration
+----------------------
+
+At some point after acceptance of the PEP, I will likely submit the
+following MIME type registration requests to IANA:
+
+* Full metadata: ``application/vnd.python.pydist+json``
+* Essential dependency resolution metadata:
+ ``application/vnd.python.pydist-dependencies+json``
+
+It's even possible we may be able to just register the ``vnd.python``
+namespace under the banner of the PSF rather than having to register
+the individual subformats.
+
+
+String methods in environment markers
+-------------------------------------
+
+Supporting at least ".startswith" and ".endswith" string methods in
+environment markers would allow some conditions to be written more
+naturally. For example, ``"sys.platform.startswith('win')"`` is a
+somewhat more intuitive way to mark Windows specific dependencies,
+since ``"'win' in sys.platform"`` is incorrect thanks to ``cygwin``
+and the fact that 64-bit Windows still shows up as ``win32`` is more
+than a little strange.
 
 
 Support for metadata hooks
 ---------------------------
 
-This feature is provided by the ``python.metadata_hooks`` extension in
-:pep:`459`.
-
-The new metadata hook system is designed to allow the wheel format to fully
-replace direct installation on deployment targets, by allowing projects to
-explicitly define code that should be executed following installation from
-a wheel file.
+While a draft proposal for a `metadata hook system
+<https://bitbucket.org/pypa/pypi-metadata-formats/src/default/metadata-hooks.rst>`__
+has been created, that proposal is not part of the initial set of standard
+metadata extensions in PEP 459.
+
+A metadata hook system would allow the wheel format to fully replace direct
+installation on deployment targets, by allowing projects to explicitly
+define code that should be executed following installation from a wheel file.
 
 This may be something relatively simple, like the `two line
 refresh <https://twistedmatrix.com/documents/current/core/howto/plugin.html#auto3>`__
@@ -1741,195 +1947,11 @@
 meaning they cannot be deferred to implicit execution on first use of the
 distribution.
 
-The metadata hook and metadata extension systems allow support for such
-activities to be pursued independently by the individual platform
-communities, while still interoperating with the cross-platform Python
-tools.
-
-Legacy packages that expect to able to run code on target systems using
-``setup.py install`` will no longer work correctly. Such packages will
-already break when pip 1.4+ is configured to use a wheel cache directory.
-
-
-Changes to environment markers
-------------------------------
-
-There are three substantive changes to environment markers in this version:
-
-* ``platform_release`` was added, as it provides more useful information
- than ``platform_version`` on at least Linux and Mac OS X (specifically,
- it provides details of the running kernel version)
-* ordered comparison of strings is allowed, as this is more useful for
- setting minimum and maximum versions where conditional dependencies
- are needed or where a platform is supported
-* comparison chaining is explicitly allowed, as this becomes useful in the
- presence of ordered comparisons
-
-The other changes to environment markers are just clarifications and
-simplifications to make them easier to use.
-
-The arbitrariness of the choice of ``.`` and ``_`` in the different
-variables was addressed by standardising on ``_`` (as these are all
-predefined variables rather than live references into the Python module
-namespace)
-
-The use of parentheses for grouping was explicitly noted to address some
-underspecified behaviour in the previous version of the specification.
-
-
-Updated contact information
----------------------------
-
-This feature is provided by the ``python.project`` and
-``python.integrator`` extensions in :pep:`459`.
-
-The switch to JSON made it possible to provide a more flexible
-system for defining multiple contact points for a project, as well as
-listing other contributors.
-
-The ``type`` concept allows for preservation of the distinction between
-the original author of a project, and a lead maintainer that takes over
-at a later date.
-
-
-Changes to project URLs
------------------------
-
-This feature is provided by the ``python.project`` and
-``python.integrator`` extensions in :pep:`459`.
-
-In addition to allow arbitrary strings as project URL labels, the new
-metadata standard also defines a recommend set of four URL labels for
-a distribution's home page, documentation, source control and issue tracker.
-
-
-Changes to platform support
----------------------------
-
-The new environment marker system makes it possible to define supported
-platforms in a way that is actually amenable to automated processing. This
-has been used to replace several older fields with poorly defined semantics.
-
-For the moment, the old ``Requires-External`` field has been removed
-entirely. The combination of explicit support for post install hooks and the
-metadata extension mechanism will hopefully prove to be a more useful
-replacement.
-
-
-Updated obsolescence mechanism
-------------------------------
-
-The marker to indicate when a project is obsolete and should be replaced
-has been moved to the obsolete project (the new ``obsoleted_by`` field),
-replacing the previous marker on the replacement project (the removed
-``Obsoletes-Dist`` field).
-
-This should allow distribution tools to more easily warn users of
-obsolete projects and their suggested replacements.
-
-The ``Obsoletes-Dist`` header is removed rather than deprecated as it
-is not widely supported, and so removing it does not present any significant
-barrier to tools and projects adopting the new metadata format.
-
-
-Included text documents
------------------------
-
-This feature is provided by the ``python.details`` extension in :pep:`459`.
-
-Currently, PyPI attempts to determine the description's markup format by
-rendering it as reStructuredText, and if that fails, treating it as plain
-text.
-
-Furthermore, many projects simply read their long description in from an
-existing README file in ``setup.py``. The popularity of this practice is
-only expected to increase, as many online version control systems
-(including both GitHub and BitBucket) automatically display such files
-on the landing page for the project.
-
-Standardising on the inclusion of the long description as a separate
-file in the ``dist-info`` directory allows this to be simplified:
-
-* An existing file can just be copied into the ``dist-info`` directory as
- part of creating the sdist
-* The expected markup format can be determined by inspecting the file
- extension of the specified path
-
-Allowing the intended format to be stated explicitly in the path allows
-the format guessing to be removed and more informative error reports to be
-provided to users when a rendering error occurs.
-
-This is especially helpful since PyPI applies additional restrictions to
-the rendering process for security reasons, thus a description that renders
-correctly on a developer's system may still fail to render on the server.
-
-The document naming system used to achieve this then makes it relatively
-straightforward to allow declaration of alternative markup formats like
-HTML, Markdown and AsciiDoc through the use of appropriate file
-extensions, as well as to define similar included documents for the
-project's license and changelog.
-
-Grouping the included document names into a single top level field gives
-automated tools the option of treating them as arbitrary documents without
-worrying about their contents.
-
-Requiring that the included documents be added to the ``dist-info`` metadata
-directory means that the complete metadata for the distribution can be
-extracted from an sdist or binary archive simply by extracting that
-directory, without needing to check for references to other files in the
-sdist.
-
-
-Appendix D: Deferred features
-=============================
-
-Several potentially useful features have been deliberately deferred in
-order to better prioritise our efforts in migrating to the new metadata
-standard. These all reflect information that may be nice to have in the
-new metadata, but which can be readily added in metadata 2.1 without
-breaking any use cases already supported by metadata 2.0.
-
-Once the ``pypi``, ``setuptools``, ``pip``, ``wheel`` and ``distlib``
-projects support creation and consumption of metadata 2.0, then we may
-revisit the creation of metadata 2.1 with some or all of these additional
-features.
-
-
-MIME type registration
-----------------------
-
-At some point after acceptance of the PEP, I will likely submit the
-following MIME type registration requests to IANA:
-
-* Full metadata: ``application/vnd.python.pydist+json``
-* Essential dependency resolution metadata:
- ``application/vnd.python.pydist-dependencies+json``
-
-It's even possible we may be able to just register the ``vnd.python``
-namespace under the banner of the PSF rather than having to register
-the individual subformats.
-
-
-String methods in environment markers
--------------------------------------
-
-Supporting at least ".startswith" and ".endswith" string methods in
-environment markers would allow some conditions to be written more
-naturally. For example, ``"sys.platform.startswith('win')"`` is a
-somewhat more intuitive way to mark Windows specific dependencies,
-since ``"'win' in sys.platform"`` is incorrect thanks to ``cygwin``
-and the fact that 64-bit Windows still shows up as ``win32`` is more
-than a little strange.
-
-
-Additional install hooks
-------------------------
-
-In addition to the postinstall and postuninstall hooks currently described
-in :pep:`459`, other distribution systems (like RPM) include the notion of
-preinstall and postuninstall hooks. These hooks would run with the runtime
-dependencies installed, but without the distribution itself. These have
-been deliberately omitted for the time being.
+For the time being, any such system is being left to the realm of tool
+specific metadata extensions. This does mean that affected projects may
+choose not to publish wheel files, instead continuing to rely on source
+distributions until the relevant extension is well defined and widely
+supported.
 
 
 Metabuild system
-- 
Repository URL: http://hg.python.org/peps
