The pkgsrc wrapper framework

EuroBSDCon 2009

The pkgsrc wrapper framework

Jörg Sonnenberger

joerg@NetBSD.org

Motivation

• Reliable packages
• Reproducable builds
• Inconsistent command lines of compilers
• Inconsistencies between Operating Systems

The buildlink framework

• buildlink3.mk files provide dependency information
• buildlink3.mk files provide the visibility tree
• Shadow tree in the work area:
  • .buildlink subdirectory with symlinks to /usr/pkg
  • .x11-buildlink subdirectory with symlinks to /usr/X11R6

The buildlink framework

• File list based on package content
• Header files, libraries, pkg-config files per default
• Wrappers to redirect to the shadow tree

Transformations for GCC-like frontends (I)

• Multiple transformation phases:
  • Normalization
  • First cleanup
  • Generic option transformation
  • Reordering
  • Compiler-specific option transformation
  • Second cleanup

Transformations for GCC-like frontends (II)

• Each phase can add, change or drop options
• Input: GCC-like command line
• Output: native compiler command line

Phase 1: Normalization (I)

• Transform options to canonical form
• "-Dmacro" vs "-D macro", "-Idir" vs "-I dir", "-Ldir" vs "-L dir"
• Linker options can be concatenated ("-Wl,-L,dir")
• "-Ldir" vs "-Wl,-Ldir"

Phase 1: Normalization (II)

• "-R" vs "-Wl,-R" vs "-Wl,-rpath" vs "-Wl,--rpath"
• "-Wl,-rpath,dir1:dir2" vs "-Wl,-rpath,dir1 -Wl,rpath,dir2"
• Same for "-Wl,-rpath-link"
• "/usr/pkg/lib/libgtk2.so" vs "-L/usr/pkg/lib -lgtk2"

Phases 2 & 6: Cleanup

• Drop redundant arguments:
  • Simplifies following phases
  • Helps platforms with small argument limit
• Duplicate "-I", "-L", "-Wl,-rpath" and "-Wl,-rpath-link"
• Old wrappers: Drop duplicate "-D" options
• Drop "-Wl,-rpath" options with relative paths
• Unify consecutive library options

Phase 3: Generic option processing (I)

• Implements the shadow tree redirection
• Supports dropping or transforming flags:
  • Optimizer flags that crash the compiler
  • Unsupportable warning flags ("-Werror")
  • Old wrapper: wildcard matches

Phase 3: Generic option processing (II)

• Absolute path names for "-I", "-Wl,-rpath" and "-L" are matched
• Per type src:dst rules
• Replace src part with dst, drops option if dst is empty
• No match: option is dropped

Phase 4: Reordering

• Helps with stricter library ordering rules:
  • Mach-O, XCOFF
  • Static linkage
• foo:bar moves all instances of foo after first instance of bar
• Old wrappers: Move all "-l" options after all "-L" arguments
• Old wrappers: Glue for "-Wl,-Bdynamic" and "-Wl,-Bstatic"

Phase 5: Platform specific processing

• Transforms GCC options to native compiler options
• Platform specific linker options
• Example AIX: emulation of "-Wl,-rpath" using "-blibpath"
• Table driven transformation of fixed strings or prefix matches

Dealing with libtool

• Compiler wrapper to abstract shared library issues
• Need to deal with full path to libtool archives (*.la)
• Relinking on installation
• Leaking references to .buildlink inside libtool archives
• Wrap libtool itself for pre and post processing

Libtool: Full path to libtool archives

• Path is in the work area: leave it
• Transform it to "-L" + "-l" argument pair

Libtool: Relinking

• Need to be able to link against to-be-installed libraries
• Need to be able to link against unshadowed libraries
• Solution: add .libs subdirectory of depending libtool archives via "-L"
• Problem: all libtool archives must be requested by path
• Solution: scan "-l" options and replace them if needed

Libtool: .buildlink leakage

• libtool includes "-L" options for linkage in libtool archives
• libtool sees only the buildlink shadow tree
• Unwrap the dependency_libs line by replacing .buildlink references back
• Only relevant for .lai files (to-be-installed version of .la)

The new wrappers

• Compiled C code
• Configuration file for each wrapper
• Most errors are fatal: no silent problems
• Stricter validation: less cross-platform issues

The new wrappers: code overview (I)

• Independent loops over all arguments for each phase
• Arguments as (mostly) immutable strings: copy-on-change
• Hash tables for the cleanup phase
• Perfect hashing for compiler transformation

The new wrappers: code overview (II)

• Forking only if needed (libtool)
• No low level optimisations yet
• NetBSD/gcc: 64KB or 2600 lines of code

Performance (I)

Performance (II)

Performance (III)

Conclusions

• Little to no overhead
• Small code base
• Straight forward logic (except libtool)
• Separation of code and configuration
• More work to finish before merge

Q&A

Questions?