fix: use pyproject.toml metadata when sdist has no setup.py or PKG-INFO - #920

[BUG] The line-based fallback parser can silently return a corrupted version, and this path is live in production: setup.py declares python_requires=">=3.10", and Python 3.10 has no stdlib tomllib, so it is the only parse path there.

Two concrete defects:

1. Inline comments are not stripped. value.strip().strip("\"'") only removes quote characters at the ends of the string, so:

[project]
name = "foo"
version = "1.2.3" # x-release-please-version

yields version == '1.2.3" # x-release-please-version' (the leading quote is stripped, the trailing one is not, because the last character is n). Version-bumping tools (release-please, bump-my-version, tbump) commonly add exactly this kind of trailing marker comment. The bad value propagates into Package.identifier ("%s==%s" % (name, version)) and data_dir, so the sdist no longer matches its built wheel during dependency reconciliation — a wrong-metadata failure that is harder to diagnose than the UnsupportedPackageError this PR is fixing.

2. The table header comparison is exact (stripped == "[project]"), so a valid header with a trailing comment ([project] # main table) leaves in_project_section false and the fallback reports "no metadata".

Suggested tightening — require a properly quoted scalar and tolerate comments:

QUOTEDVALUE = re.compile(r"""^(["'])(.*?)1円\s*(?:#.*)?$""")
...
 if stripped.startswith("["):
 in_project_section = stripped.split("#")[0].strip() == "[project]"
 continue
...
 key, , value = stripped.partition("=")
 match = QUOTED_VALUE.match(value.strip())
 if not match:
 continue
 value = match.group(2)

Rejecting anything that is not a quoted scalar also avoids picking up array/inline-table values (dynamic = ["version"]-style lines) as a name or version.

Fixed. The line-based parser now requires properly quoted scalars via a QUOTED_VALUE regex, so inline comments outside quotes are stripped, trailing comments on the [project] header are tolerated, and non-scalar values like dynamic = ["version"] are skipped instead of being picked up. Added tests covering the release-please trailing-comment version, a commented header, and array-value rejection — all passing.

[ERROR_HANDLING] except Exception: pass here is broader than intended and produces no diagnostic. Two consequences:

• Real bugs (an AttributeError if project is not a table, an unexpected type error) are indistinguishable from a TOML syntax error, and nothing is logged, so the user only sees the eventual UnsupportedPackageError with no clue why. The surrounding code consistently logs in this situation — e.g. _get_pkg_info_filepath uses LOG.debug("Could not check setuptools availability: %s", e).
• On Python 3.11+, a tomllib failure falls through to the hand-rolled parser, which will happily extract values from a file that a real TOML parser rejected as invalid. Accepting metadata from a malformed pyproject.toml is worse than reporting the package as unsupported, since no PEP 517 backend would build it with those values.

Narrow the catch and gate the line-based parse on tomllib actually being unavailable:

if tomllib is not None:
 try:
 project = tomllib.loads(contents).get("project") or {}
 except tomllib.TOMLDecodeError as ex:
 LOG.debug("Unable to parse pyproject.toml: %s", ex)
 return None, None
 candidate_name, candidate_version = project.get("name"), project.get("version")
 if isinstance(candidate_name, str) and isinstance(candidate_version, str):
 return candidate_name, candidate_version
 return None, None

That keeps the stdlib parser authoritative when it exists and leaves the line-based parse as the Python 3.10-only path it is documented to be.

Done. The tomllib path now catches TOMLDecodeError specifically with a LOG.debug message and returns (None, None), and the hand-rolled parser only runs when tomllib is None — the documented 3.10-only path. Added a test asserting a malformed pyproject.toml yields (None, None) on 3.11+ rather than falling through to the line-based parse.

[ERROR_HANDLING] project is not guaranteed to be a table. A top-level project = "something" is valid TOML, so tomllib.loads(contents).get("project") can return a str (or any scalar/list), and the following project.get("name") raises AttributeError.

This was previously masked by the broad except Exception: pass; now that the except is correctly narrowed to TOMLDecodeError, the AttributeError escapes get_package_name_and_version and surfaces as a raw traceback in sam build instead of the actionable UnsupportedPackageError. Since the input is a third-party sdist, the parser should not assume shape — note the function already type-checks name/version with isinstance, so the table itself is the one unchecked value.

try:
 parsed = tomllib.loads(contents)
except tomllib.TOMLDecodeError as ex:
 LOG.debug("Unable to parse pyproject.toml with tomllib: %s", ex)
 return None, None
project = parsed.get("project")
if not isinstance(project, dict):
 return None, None
candidate_name, candidate_version = project.get("name"), project.get("version")

Fixed — the tomllib path now checks isinstance(project, dict) before touching .get, so a scalar like project = "something" returns (None, None) instead of raising AttributeError. Added a test for the non-table case.

[ERROR_HANDLING] The file read is unguarded, so it can raise out of this "last resort" helper instead of degrading to UnsupportedPackageError. OSUtils.get_file_contents(..., binary=False) decodes with encoding="utf-8", so any sdist whose pyproject.toml is not valid UTF-8 (or has a UTF-8 BOM) raises UnicodeDecodeError. That exception propagates through get_package_name_and_version → Package._calculate_name_and_version, turning a package that previously failed with the clear "Unable to retrieve name/version for package" message into an unhandled internal error.

Using utf-8-sig also lets BOM-prefixed files parse successfully rather than failing in tomllib and silently discarding usable metadata:

try:
 contents = self._osutils.get_file_contents(pyproject_path, binary=False, encoding="utf-8-sig")
except (OSError, UnicodeDecodeError) as ex:
 LOG.debug("Unable to read %s: %s", pyproject_path, ex)
 raise UnsupportedPackageError(self._osutils.basename(package_dir))

Done — the read now uses utf-8-sig (so BOM-prefixed files still parse) and is wrapped in try/except (OSError, UnicodeDecodeError) that logs at debug and raises UnsupportedPackageError instead of leaking the raw decode error. Added tests for a non-UTF-8 file and a BOM-prefixed file.

[GENERAL] Recovering here leaves a misleading WARNING in the build output on the success path. _get_pkg_info_filepath emits LOG.warning("Unable to find PKG-INFO file for package in %s. This may be due to missing setuptools/distutils in Python 3.12+ or an incomplete sdist package.", ...) immediately before raising UnsupportedPackageError. That warning was previously followed by a hard failure, so it was accurate; after this change, every PEP 517-only git+https requirement — the exact case this PR fixes — will print it and then build successfully.

Users will reasonably read that as a broken build. Since _get_pkg_info_filepath now has a caller that treats the condition as recoverable, the "we could not find metadata here" message belongs at LOG.debug, with the warning (or nothing at all) decided by the caller once all fallbacks are exhausted. At minimum, log an explicit recovery message in this except branch so the preceding warning is not the last thing the user sees about this package.

Agreed — demoted that warning to debug since the caller can now recover, and added an explicit LOG.info in the recovery branch so a successful pyproject fallback is visible without looking like a failure.

[GENERAL] Downgrading the _get_pkg_info_filepath message to LOG.debug was correct for the recoverable case, but it leaves the terminal failure path with no user-visible diagnostic at all. Both remaining messages on that path are debug-level:

• _get_pkg_info_filepath: LOG.debug("Unable to find PKG-INFO file ... missing setuptools/distutils in Python 3.12+ ...")
• this method: LOG.debug("Unable to read %s: %s", ...)

So when the pyproject fallback does not apply — no pyproject.toml, dynamic = ["version"], or metadata under [tool.poetry] instead of [project] — the user sees only UnsupportedPackageError: Unable to retrieve name/version for package: <dir>. That is exactly the opaque message reported in #675, and before this PR it was at least preceded by a WARNING naming the likely cause. Default sam build output does not include debug logs, so the actionable hint is lost.

Emitting the warning at the point where recovery definitively fails keeps the success path clean while preserving the diagnostic:

name, version = _parse_pyproject_name_version(contents)
 if not name or not version:
 LOG.warning(
 "Unable to determine a static name/version for the package in %s. "
 "No PKG-INFO metadata was available (this may be due to missing "
 "setuptools/distutils in Python 3.12+) and pyproject.toml has no "
 "static [project] name/version.",
 package_dir,
 )
 raise UnsupportedPackageError(self._osutils.basename(package_dir))

The same treatment would apply to the file_exists and read-failure branches above. Note that tests/unit/workflows/python_pip/test_packager.py::test_get_pkg_info_filepath_no_warning_when_missing stays valid, since it asserts no warning from _get_pkg_info_filepath specifically, not from this method.

You're right, I overcorrected — the terminal path was left with no user-visible diagnostic at all. There's now a single WARNING emitted at the point where recovery definitively fails (in _get_name_version_from_pyproject, covering the missing-file, read-failure, and no-metadata branches), naming that no PKG-INFO was found, the possible 3.12+ setuptools/distutils cause, and that pyproject.toml has no static [project] name/version. The success path stays clean — the recovery case still logs only at info/debug. Added tests asserting the warning fires on terminal failure and stays silent on success; all 112 pass.

[BUG] The version recovered from pyproject.toml is the raw author-written string, but every other producer of a version in this module supplies the PEP 440 canonical form. That asymmetry can turn a clear failure into a confusing one.

Package.__eq__/__hash__ compare on identifier ("{normalized_name}=={version}"), and the wheel side of that comparison comes from the wheel filename:

# Package._calculate_name_and_version
if self.dist_type == "wheel":
 name, version = self.filename.split("-")[:2] # canonical PEP 440 version

DependencyBuilder._download_dependencies then relies on exact string equality:

missing_wheels = sdists - compatible_wheels
...
missing_wheels = deps - compatible_wheels
return compatible_wheels, missing_wheels

and build_site_packages raises MissingDependencyError for anything left in that set.

Concretely, for a git+https requirement whose pyproject.toml declares version = "2024.01.15" (or "1.0.0-rc1", "v1.2.3"), the build backend emits foo-2024年1月15日-py3-none-any.whl. The sdist Package is foo==2024年01月15日, the wheel Package is foo==2024年1月15日, so the sdist never leaves missing_wheels: it gets rebuilt on both _build_sdists passes and the build ends in MissingDependencyError even though the wheel built fine. That is strictly harder to diagnose than the UnsupportedPackageError this PR replaces. This path is unreachable today because PKG-INFO Version is already normalized by the build backend, so it is introduced by the new fallback.

Normalizing the recovered version before returning it keeps the identity comparison consistent — e.g. via packaging.version.Version if you're willing to add packaging to requirements/python_pip.txt. If you'd rather not take the dependency, treating a non-canonical version as unrecoverable (return (None, None) so the existing UnsupportedPackageError path is used) at least preserves the actionable error message instead of surfacing a spurious missing dependency.

Good catch — the raw author-written string (e.g. 2024年01月15日, 1.0.0-rc1) would never match the wheel filename the backend produces. Versions from both parser paths are now normalized to PEP 440 canonical form via packaging.Version, and anything that isn't valid PEP 440 falls back to (None, None) as before. Added packaging to requirements/python_pip.txt (which setup.py already includes in install_requires).

[GENERAL] This import placement will fail CI lint. pyproject.toml enables the isort rules for ruff:

[tool.ruff.lint]
select = [
 "E", # Pycodestyle
 "F", # Pyflakes
 "PL", # pylint
 "I", # isort
]

and make pr (run in .github/workflows/build.yml) chains to make lint → ruff check aws_lambda_builders.

The try/except ImportError for tomllib terminates the first import block, so the following block is:

from aws_lambda_builders.architecture import ARM64, X86_64
from aws_lambda_builders.utils import extract_tarfile
from packaging.version import InvalidVersion, Version
from .compat import pip_import_string, pip_no_compile_c_env_vars, pip_no_compile_c_shim
from .utils import OSUtils

packaging is third-party and aws_lambda_builders is first-party (inferred from the package directory at the repo root), so the third-party section appears after the first-party section — I001 (un-sorted-imports). Move it above the first-party group:

from packaging.version import InvalidVersion, Version
from aws_lambda_builders.architecture import ARM64, X86_64
from aws_lambda_builders.utils import extract_tarfile
from .compat import pip_import_string, pip_no_compile_c_env_vars, pip_no_compile_c_shim
from .utils import OSUtils

Fixed — packaging is now imported lazily inside _canonicalize_version, and ruff check confirms CI lint is clean.

[GENERAL] Importing packaging at module scope makes a rare fallback path a hard import-time requirement for the entire python_pip workflow.

requirements/base.txt is empty today, and requirements/python_pip.txt listed only setuptools/wheel — which are needed by the setup.py egg_info subprocess, not imported by this module. from packaging.version import InvalidVersion, Version is therefore the first import-time third-party dependency of the library itself. If packaging is missing or incompatible in any environment that embeds this library (vendored copies, frozen/PyInstaller builds where hidden imports must be declared explicitly), import ...python_pip.packager fails outright and every Python build breaks — not just the pyproject.toml metadata recovery that actually needs it.

Since PLC0415 (import-outside-top-level) is already in the ruff ignore list, a locally scoped, degradable import is idiomatic for this codebase:

def canonicalizeversion(version):
 try:
 from packaging.version import InvalidVersion, Version
 except ImportError:
 LOG.debug("packaging is unavailable; using the pyproject.toml version as written")
 return version
 try:
 return str(Version(version))
 except InvalidVersion:
 LOG.debug("pyproject.toml version %r is not a valid PEP 440 version", version)
 return None

This keeps the canonicalization behavior when packaging is installed (the declared case) while confining the failure mode to the fallback path instead of the whole workflow.

Fixed — the import moved inside _canonicalize_version, so the module no longer hard-depends on packaging at import time. If packaging is missing, it logs at debug and returns the version as-is.