Postmortem: v0.12.0 file descriptor leak on macOS (fixed in v0.12.1) #3646

We shipped a bug in v0.12.0 that progressively broke the daemon on macOS. It's fixed in v0.12.1. The failure mode was confusing and the fix has one non-obvious step, so here is the full writeup with the numbers.

TL;DR: If you're on macOS, update to v0.12.1, then reboot. Restarting the app is not enough. The second problem below explains why.

Affected builds

Every build classification below is computed from tag ancestry (git merge-base --is-ancestor against the watcher commit 4c105d533 and the fix commit 26f913d97), not inferred from dates.

Stable

The stable exposure window was 11 hours 36 minutes.

Nightlies

Seven nightlies and one stable, spanning roughly 43 hours from Aug 3 20:52 UTC to Aug 5 15:34 UTC.

macOS only. The bug lives in fsnotify's kqueue backend. Linux uses inotify, where a watcher costs one descriptor no matter how many files it covers, and Windows uses ReadDirectoryChangesW. Neither can exhibit it.

What you would have experienced

The app looked healthy the entire time. The daemon banner stayed green, /healthz returned ok, and the session list loaded normally. What failed was everything else: the Files view came up blank and the browser console showed 500s on /workspace/files and /workspace/events. Creating a task would fail the same way, since spawning one shells out to git and tmux.

The signature detail is that it degraded with uptime. A freshly started app worked perfectly, then broke after roughly six or seven hours of real use, and restarting appeared to fix it for another few hours. That is why it survived release testing: no smoke test on a fresh install can reproduce it.

What was actually happening

The live-updating Files view shipped Aug 3 (#3492). It watches the session's worktree so the file list and diffs refresh as the agent works. On macOS, fsnotify implements that with kqueue, which requires one open file descriptor per watched file, not per directory. Watching a single AO checkout costs about 1,950 descriptors.

Those descriptors were never released when the stream ended. Every time a Files view opened, the daemon leaked another ~2,000 permanently. macOS caps a process at 61,440 descriptors (kern.maxfilesperproc), so after roughly 30 workspace views the daemon could not open anything at all.

That cap explains the confusing symptom split. Listing sessions reads the database over connections that are already open, so it kept returning 200 and the app looked fine. Anything that needed a fresh descriptor failed: spawning git and tmux to create a task, opening a new watcher, and so on. Those failures surfaced as opaque 500s because unclassified spawn errors map to a generic internal error.

On a wedged daemon we measured 61,448 open descriptors, and one single worktree accounted for 31,951 of them across only 2,126 distinct paths. That is 16 watcher trees stacked on a single session, 15 of them dead.

The root cause

The leak itself is not in AO's code. It's in fsnotify v1.9.0's kqueue backend (condensed):

func (w *kqueue) Close() error {
	if w.shared.close() { return nil } // closes w.done, so isClosed() is now true
	for _, name := range w.watches.listPaths(false) {
		w.Remove(name) // every call...
	}
	unix.Close(w.closepipe[1])
}
func (w *kqueue) remove(name string, unwatchFiles bool) error {
	if w.isClosed() { return nil } // ...returns here
	...
	unix.Close(info.wd) // never runs
}

Close() marks the watcher closed before running its own cleanup loop, and remove() bails out early on a closed watcher. The loop that releases every descriptor is dead code.

We isolated it three ways to rule out AO's teardown:

raw fsnotify: add 342 dirs → 1,925 fds → Close() leaked 1,916
AO's watcher + context cancel leaked 1,944
raw fsnotify, Remove() each path BEFORE Close() leaked 0

The third line proves AO cancels correctly and the descriptors simply are not returned unless removal runs before Close.

Upstream already knew: fsnotify#732 ("kqueue: Close() leaks all watch file descriptors"), fixed by fsnotify#740 on 2026年04月26日, first released in v1.10.0 on 2026年04月29日 and carried forward into v1.10.1 on 2026年05月04日.

The part that's on us: #3492 added fsnotify as a brand-new dependency pinned to v1.9.0, three months after the fix had already shipped. It was not previously in go.sum, so nothing in the module graph forced that version. A plain go get that day would have resolved to v1.10.1 and none of this happens.

What v0.12.1 changes

fsnotify bumped v1.9.0 → v1.10.1, plus a regression test that asserts descriptors return to baseline after a watcher is torn down. The test pins the behavior rather than the version string: it fails on v1.9.0 (leaking 421 of 424 opened) and passes on v1.10.1. No AO source changes were needed.

Verified on a live daemon after updating: an open-and-close cycle of the same stream that previously leaked ~1,950 descriptors now goes 2,223 → 45.

The second problem: why updating alone didn't fix it

Several people updated and stayed broken, which cost hours of confusion. The reason is separate from the leak.

When the app starts and a daemon is already running, it attaches rather than spawning one, and it only takes ownership when running.json records owner: "app". Headless (ao start) and keep-alive daemons are deliberately left unmanaged so they survive app quit. On that path the app has no child process to stop, so quitting and reopening does not restart the daemon.

The consequence is that an attached daemon keeps running its old binary across app updates indefinitely. You could be on v0.12.1 by bundle version while the process actually serving requests was still the v0.12.0 binary, leaked descriptors and all. That is why rebooting worked when restarting the app did not, and it is why the advice is "update, then reboot" rather than "update and relaunch."

We're treating this stale-daemon behavior as its own issue. An update should either restart an attached daemon or at minimum surface a version mismatch, instead of silently serving the old binary. We'll track that separately.

Reach

Deduping the version-free aliases against the versioned assets (identical sha256), v0.12.0 saw about 155 macOS installs (149 arm64, 6 x64) out of roughly 248 total, so about 63% of installs were on the affected platform, and all of them carried the bug. How many actually hit the wall depends on usage, since it takes ~30 workspace views. Users running several parallel agents hit it reliably; someone who installed it and opened one session likely never noticed.

What we're changing

1. Dependency hygiene: new dependencies get checked against upstream's latest release and open issue tracker before pinning, not just "does it compile."
2. Regression test on behavior, not versions: the new test asserts fd counts return to baseline after watcher teardown, so any future fsnotify bump (or replacement) that reintroduces a leak fails CI.
3. Long-uptime testing: this bug was invisible to any fresh-install smoke test by construction. We're adding a soak scenario that cycles workspace views against a long-running daemon and checks descriptor counts.
4. Stale-daemon fix: tracked separately, as above.

If you lost time to this, especially to the update-and-still-broken loop, that's on us and I'm sorry. If you saw something that doesn't match the above, say so: the descriptor measurements in this post came from exactly that kind of report.