lochley.NetBSD.org

• Former home of cvs.NetBSD.org
• Lots of RAM, lots of disk space
• New home of the automated CVS -> Fossil -> Git conversion
• ...and now home of the CVS -> Fossil -> Mercurial conversion, too

lochley and Mercurial: challenges

• fastimport used one directory with one file per blob
• ...with uncompressed content
• Duplicate commit check happens after transaction locks
• fastimport assumed markers are persistent
• ...tagging commits with markers, breaking duplicate detection
• fastimport missed support for its own mark file
• ...no incremental conversion without fixing it

lochley and Mercurial: status quo

• Conversion fully integrated now
• Similar time to the git cycle
• Results pushed to Bitbucket
• ...beware that src is pushing some scalability limits for BB

Mercurial performance relative to Git (I)

• Basic testing on E5 E3-1245v5 under Illumos with 100Mbit/s+ connectivity
• hg init + unbundle from cdn.NetBSD.org: ~8min, CPU bound (zstd)
• ...with zstd as revlog compression engine: 6min
• hg clone from https://bitbucket.org/netbsd/src: 10min (bzip2)
• ...with zstd as revlog compression engine: 7min
• git clone: ~5min with 3min local CPU time
• git clone needs 50% more bandwidth than the bundle

Mercurial performance relative to Git (II)

• hg co -r trunk after initial clone: 40s (2min CPU time)
• hg co -r netbsd-8 and back to trunk: 6s / 8s (16s / 22s CPU time)
• git checkout netbsd-8 and back to trunk: 5s / 8s (6s / 8s CPU time)
• hg commit vs hg commit BUILDING: 2s / 0.8s
• git commit -a vs git commit BUILDING: 0.7s / 2s
• hg blame UPDATING: 0.7s
• git blame UPDATING: 2s

Check list: compared to FreeBSD's VCS wiki

• CVS migration: solved
• History removal: possible, but tricky with convert
• Base system integration: not planned
• Database recovery: double check transaction assumptions
• Partial clones: needs investigation, shallow and remotefilelog extensions

Check list: core@ requirements (I)

• Performance: testing on low-end hardware in progress
• ...but comfortable on the 90$ Pinebook under Linux
• ...peak RSS on Linux ~800MB
• User interface and terminology mostly compatible with CVS
• Concurrent use of CVS: not planned
• ...but preparation and testing of infrastructure against Mercurial
• ...expected down time for final migration: around 2 days

Check list: core@ requirements (II)

• hg ↔ git bridges exists, performance and limitations need to be checked
• Automatic clean-up of CVS revision references: not feasible
• Repository conversion: character set conversion in CVS before final migration
• ...no detection of moves/copies planned
• ...no retro-patching of SCCS history planned

Pending work (I)

• Provide documented workflows for common operations, using modern best practises
• ...include merge vs rebase discussion for local changes
• ...staging repository vs "core" repository
• Work out the build and testing infrastructure
• ...move towards more off-the-shelf components
• ...somewhat easier with changesets
• ...currently under active investigation

Pending work (II)

• Figure out new release engineering workflow
• ...easier with changesets
• Current approach: push changes to staging repository
• ...CI verifies building (and maybe passing tests)

Pending work (III)

• Test repository server
• ...automatic pushing of clone bundles to ftp.NetBSD.org
• ...test build host attached
• ...allow pushing commits for authentication testing
• ...provide source-changes alternative based on hg notify
• Investigate Security Team ↔ Releng interaction

Q&A