pbulk

Jörg Sonnenberger

joerg@NetBSD.org

Overview

Background and Motivation
Anatomy of a bulk build
pbulk components
Future plans

Some background

Fixing issues in FreeBSD ports since end of 2003
Moved to pkgsrc in summer 2005
Regular bulk builds since October 2005
Rule: A problem fixed in the bulk build is a problem no user can complain about.
Rule: A problem in a bulk build is easier to reproduce than a problem from a user.
Binary packages are a by-product

Anatomy of a bulk build

Setup of machine and bulk build
Prepare the pkgsrc tree
Find out what to build
Build packages
Summarize results
Upload results
Upload binary packages

Specific problems

Stray directories in /usr/pkg
Stray dependencies in /usr/pkg (gmake, pkg-config)
.if exists(${X11BBASE}/include/X11/Xdefs.h) ... .endif
math/p5-Math-Pari
Writing to /etc
Writing to /home
Scalability

pbulk components (I)

pbulk-scan: Schedule scanning of pkgsrc parts
pbulk-resolve: Compute dependency tree
pbulk-build: Schedule build of packages
Small glue in mk/pbulk for scan target

pbulk components (II)

Schedulers in C: network access, memory usage
Resolver in C: code reuse for Dewey, speed
Upload logic somewhat overlapped with pbulk-build
Report logic completely in awk

The scan component

Take list of PKGPATH or extract subdirectories using make
Pull/push model for distributed scans
Call "make pbulk-index" for every PKGPATH
make is the only way to get reliable results
Collect and reorder list, drop duplicate entries

The resolver component

Implement a mix of pkg_add and pkgsrc algorithm to find dependencies
If in doubt, prefer pkgsrc view, but warn
{gs,gs-esp}-[0-9]*:../../print/ghostscript requested, gs-7.0 and gs-esp-7.0nb1 exist — what to pick?
Complain or list what couldn't be found

The build component

Pull/push model for distributed scans
Find package with all dependencies done and highest priority
Sort order: number of packages depending on this

Future plans

Allow to easier restrict multi-version logic
Allow using incremental mode + post-filtering when missing dependencies
Move restricted subset logic for upload into upload itself
Tag bootstrap packages and don't install them
Remove package if install/deinstall test fails
Implement poll loop directly to get rid of libevent
Split into base part and distinguish official bulk build and simple user case

Q&A

Questions?