EuroBSDCon • 11 November 2006

The redesign of pkg_install for pkgsrc

The redesign of pkg_install for pkgsrc

Jörg Sonnenberger

joerg@pkgsrc.org

The NetBSD Project

Outline

• pkgsrc in 3 minutes
• The old pkg_install
• Package metadata
• Package APIs
• Integration and conversion
• Outlook

pkgsrc in 3 minutes

• Framework for building third party software
• Cross platform
• System of choice on NetBSD and DragonFly
• Around 6300 packages
• Spun off from FreeBSD ports in 1997

pkgsrc: high-level structure

• Build descriptions (Makefiles)
• Common build rules
• buildlink3/wrapper framework
• pkg_install

What is pkg_install?

• pkg_add
• pkg_delete
• pkg_create
• pkg_info
• pkg_admin
• ...

The old pkg_install

• Inherited from FreeBSD
• Extended over the time to add some feature
• Grown code

pkg_install — illustrated

spaghetti (from Wikipedia)

Problems en detail

• Use of external programs during extraction and removal
• Code full of side effects
• Temporary copy during extraction
• Missing, outdated or incomplete specs
• Redundancy of scripts
• Interaction with high-level tools

Pkgsrc: open issues

• Advanced update facilities
• Coherent behaviour between builds from source and binary packages

The answer

• tech-pkg@NetBSD.org, April 2006: Google SoC
  Evaluation of the Status Quo
• Paris, May 2006: pkgsrccon
  Detailed requirements for a new pkg_install

Desired structure

onion (from Wikipedia)

Package patterns (I)

Before

• Plain package names: exact match
• Dewey patterns: gdm>=2.14<2.14.8
• Fnmatch patterns: pear-5.0.[0-9]*
• Csh-style alternatives: sun-{jre,jdk}<1.3.1.0.2

Problems with old patterns

• Which match is better?
• How ambigious is a rule?
• What does gdm<2.14>2.14.8 mean?

Package patterns (II)

After

• Dewey-like atoms:
  • package base name
  • operator, version pairs
  • relation operator like >=, >, ==, ...
  • prefix match operator ~
• Alternatives using |
• Explicit ordering rules

Use of patterns

• Dependencies
• Conflicts
• Vulnerabilities
• Compatibility

Compatibility patterns

• APIs are forward compatible until broken: open dependencies
• ABIs are backward compatible unless broken: closed dependencies
• Problem: point of breakage not known in advance
• Solution: packages specify what they are compatible to
• Back-port planned

Package lists

• No more @exec/@unexec
• Automatic tracking of shared and intermediate directories
• Added first-class support for configuration files
• ...and configuration directories
• ...as well as files and directories under package control

Plists as mini IDS

• Checksums are stored in the Plist
• ... and so are ownership
• ... and permissions
• Tagging of entries for type-specific (de)install hooks

Regeneration and verification of installed packages

Package metadata

• Name and version of the package
• Prefix of the package
• Who maintains the package
• Target OS version and architecture
• Redistribution license
• Descriptions, possible localized

Serialized descriptions

--- %YAML:1.0 !pkgsrc.org,20060813/pkg 
pkgname: mad123-0.8.5nb2
prefix: /usr/pkg
platform: 
 os: NetBSD
 os_version: 3.99.22
 architecture: i386
 abi: 32
depends: 
- curl>=7.15.3
- libmad>=0.15.1bnb1
plist: 
 /usr/pkg/bin/mad123: !pkgsrc.org,20060813/plist/file 
 checksum: 
 RMD160: 4b8fd3e645386f1d674603a349613075d2d2377c
 modification_time: 2006年11月07日 08:16:51
 size: 20776
 permission: 0555
 /usr/pkg/man/man1/mad123.1: !pkgsrc.org,20060813/plist/file 
 checksum: 
 RMD160: 6b6a1800c664a7940975ee634b050e2ecc5397c4
 modification_time: 2006年11月07日 08:16:51
 size: 5518
 permission: 0444
meta: 
 descriptions: 
 en_US: |
 mad123 is a simple command line MP3 player using Robert Leslie's
 libmad fixed-point MP3 decoder library.
 ...
 comments: 
 en_US: Command-line MPEG audio player
 license: GPL
 maintainer: pkgsrc-users@NetBSD.org
 build_time: 2006年11月08日 14:19:12

Binary package format

• Compressed tarballs
• Strictly ordered
• Top-level index file
• Signature
• package name / subpath for each file in the package

Architecture of pkg_install

• Pattern API: matching, ordering, extraction of package names
• Package description API: accessors for all metadata, verification
• Plist API: new plist entries, iteration, verfication
• Database API: querying, regeneration, modification groups

Integration I: staged installation

• Direct installation to /usr/pkg considered evil:
  • Hard to trace implicit changes
  • No protection against side-effects
  • Correctness of binary packages
• Instead: installation to temporary directory
• Optionally: full package building as normal user
• Utilizes WRKDIR reference check
• Conversion in progress, 50% mark is aim for pkgsrc-2006Q4

Integration II: pattern conversion

• Most nasty wildcard patterns already fixed before pksgrc-2006Q2
• Dewey patterns stay valid
• Csh-style alternatives are expanded and converted independently
• Rest converted with heuristics: 30 exceptions in pkgsrc-2006Q2

Integration III: new pkg flavour

• Callers of pkg_install mostly isolated in mk/flavour/pkg
• Conversion of mk/pkginstall to use tagging and new metadata
• Work-in-progress, as the command interfaces are not finalized

Integration IV: Old packages

• Basic metadata conversion works
• ...and is used for testing the tools
• Problem: conversion of old install scripts
  • mk/pkginstall fragments can be reasonably detected and handled
  • other fragments are difficult, no clear delimiter

Summary

• The new pkg_install:
  • Self-contained tool chain
  • Modular architecture
  • In-place operation
  • Work-in-progress
• Visible changes in pkgsrc:
  • Fixed many bogus dependencies
  • Staged installation
  • On-going efforts to annote special permissions

Q&A

Questions?