Proposal for binary package QA and upload process

To: tech-pkg%NetBSD.org@localhost
Subject: Proposal for binary package QA and upload process
From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
Date: 2024年7月15日 00:31:13 +0000

I retract my previous proposal in the thread `python woes in netbsd-10
bulk builds' and replace it by the following, which doesn't have any
reference to bulk-test-essential and doesn't hinge on the definition
of `well-defined'.
Thoughts?
1. Change
 /pub/pkgsrc/packages/NetBSD/<arch>/<version>/All
 to be a 302 Found redirect (via .bzredirect) to
 /pub/pkgsrc/packages/NetBSD/<arch>/<version>/<buildid>/All
 where <buildid> is a pbulk-chosen build id, like 20240708.0436, and
 <version> is (e.g.) 10.0 or 10.0_2024Q2 or whatever.
 Once a build has been uploaded, no changes to any of the files in
 it.
 This way, we eliminate any issue of stale CDN caches that we've
 constantly inflicted on ourselves by abusing symlinks, and any
 issue of mixing stale packages that have begun to fail to build
 with fresh packages that are incompatible.
 => To reduce storage on the server, we can use hard links via
 `rsync --link-dest=../../<prevbuildid>/All'.
 => Or, to reduce storage and bandwidth, we can set up redirects for
 packages that haven't changed so the CDN will continue to use
 the old and still-valid cache.
 (If this has to remain compatible with ftp to the same paths, we
 can jiggle things around another way, like having
 /pub/pkgsrc/packages/NetBSD/<arch>/<version>/All
 be a symlink to ../build/<version>/<buildid>/All, and having
 /pub/pkgsrc/packages/NetBSD/<arch>/<version>/.bzredirect
 point to ../build/<version>/<buildid>.)
2. Include either
 (a) the bulklog directory, or
 (b) just the bulklog/meta/ directory, or
 (c) at least a redirect to the bulk report
 in the package upload at, say,
 /pub/pkgsrc/packages/NetBSD/<arch>/<version>/<buildid>/bulklog
 (It looks like bulklog/ is usually a few hundred megabytes, so it's
 not that space-intensive on top of the whole bulk build. But maybe
 just a redirect to the builder's own URL is good enough.)
 This way, when examining a particular binary package set to see
 what went wrong, it is easy to find the report for exactly that
 build -- no need to guess at which message to pkgsrc-bulk might
 correspond with which binary packages, if any.
3. Have a program to update the .bzredirect (and log the update) only
 if it passes certain QA criteria, such as:
 (a) All/pkg_summary.gz exists and is newer than all of the packages
 (b) All/SHA512 exists and is newer than pkg_summary.gz
 (c) bulklog/meta/success exists and each package listed is in All/
 (d) bulklog/meta/report.bz2 exists is newer than All/SHA512
 (e) a message to tech-pkg@ or similar asking which broken packages
 are OK to leave broken leads, via an inevitable abstract policy
 debate, nowhere, and a coin toss comes up heads because a
 decision has to be made anyway
 and otherwise sends an alert to a low-volume list that people pay
 attention to.
4. Disable the cron job that regenerates pkg_summary, because the bulk
 builders already generate it and this is an unnecessary potential
 source of inconsistency.

Prev by Date: "up to date at all costs" is a failed approach
Next by Date: daily pkgsrc CVS update output
Previous by Thread: "up to date at all costs" is a failed approach
Next by Thread: bmake updated to 20240711
Indexes:

Home | Main Index | Thread Index | Old Index