Enable bounded Ubuntu long-soak evidence - #82

The diagnostic paid for itself in one round

At the pushed head fa0fcb12be, on Ubuntu 22.04, in a clean native worktree.

One round ago the refusal said only:

source stack did not reach the exact four-role startup cut

Four separate conditions had to hold and the message named none of them. It now
says:

source stack did not reach the exact four-role startup cut;
missing: roles;
_load_roles refused: ValueError: unclassified descendant process

That is the third time on this branch that fixing the diagnostic before
chasing the cause produced the cause immediately. The first was the engine's
RuntimeError with no message; the second was the whole runner exiting 1 with
zero bytes on both streams.

What it means

_load_roles walks the launcher's process tree and requires every descendant to
be one of the four known roles. There is a fifth process it cannot classify.

From the launcher log the tree contains at least the launcher, the ZMQ bridge
subprocess, the engine subprocess and cryodaq-assistant. So something else is
there.

Why this matters beyond the soak: a process the launcher cannot classify is
a process it cannot settle or kill. A week-long run that leaks one per restart
accumulates them, and that is the memory-and-descriptor growth the whole
endurance effort exists to rule out. If the check is right, the fix is
ownership, not the check.

Also visible in the same log, and not yet a finding

ERROR │ cryodaq.launcher │ Launcher shutdown owner remains unsettled: engine
 (RuntimeError: engine process has not yet exited after a verified
 shutdown receipt; launcher remains in HOLD pending exact process exit)

The launcher receives its shutdown receipt from the engine and the engine
process still does not exit. That may be the same story as the unclassified
descendant or a separate one; it is recorded here so it is not lost, and it will
be judged on evidence rather than folded into the current round.

What is running now

A worker names the unclassified process — identifier, parent, and command line,
because "there is a process" is not actionable — then reads how a role is
matched and whether any allow-list exists. It is told to add nothing to an
allow-list in this round, to state its hypothesis AS a hypothesis, and to say
what would settle it.

The plausible sources, none established: a Qt platform helper on a headless
host; a shell interposed by the spawn path; a multiprocessing helper from the
start method this branch changed one round ago; or a genuine leak.

@codex review

Head is 5c1335b791180ea607613c0543f63ef6acfab935. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@codex review

Head is 29c6e3ea0fbe0c394800bfbabb54ce3460d336b2. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@codex review

Head is 626be2e07bb963711de597930489742bb898adbf. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

The soak found a real endurance defect: a faulted engine never comes back

Measured at the pushed head 626be2e07b, on Ubuntu 22.04 in a clean native
worktree. The refusal now names its subject, and the first run with the named
messages produced this:

engine pid 669 faulted at elapsed 185.0s (scheduled 185.0s)
did not recover within the reviewed ceiling (60.0s):
faulted process exited and no replacement ever appeared

This is the first finding on this branch that is about the product rather than
about the soak's own machinery. Every barrier before it — the missing theme
pack, the dead safety pattern, the incomplete alarms document, the bridge
topology, the unclassified resource tracker — was scaffolding around the
measurement. This is the measurement working.

What was measured

The soak killed the engine on schedule at 185 seconds. It exited. Nothing
started another one, and sixty seconds later the run gave up.

In the same run, the bridge was faulted and did come back:

ZMQ bridge replacement committed; reason=data-flow generation=1 restart_count=2 pid=1855

So replacement exists for one role and either does not exist, or did not fire,
for the other.

What it costs

The objective is a week of continuous thermal-conductivity running without data
loss. That means the stack survives what kills a process: a crash, a kill, an
out-of-memory event. If the engine cannot come back, the run ends at the first
one, and every measurement after it is lost.

This is the roadmap's own crash/restart/election evidence item under the
pre-lab milestone, failing on the target platform, with the evidence in hand
rather than argued.

Also in the log, and possibly the same story

engine incarnation lacks exact shutdown settlement; launcher remains in permanent HOLD

If a killed engine puts the launcher into a permanent hold by design, then the
design and the week-long objective disagree — and that disagreement, not the
hold, is the finding. It will not be resolved by weakening the hold.

What a worker is doing

Establishing, with lines, whether an engine replacement path exists at all;
whether it exists and did not fire; or whether it exists and deliberately
refuses. Those are three different answers and only the middle one is a small
fix.

It is explicitly told to STOP and report rather than improvise if this turns out
to be a missing capability. Restoring a role the system already claims to own is
not a new feature; building an election protocol would be, and that is not
something to invent inside a soak-readiness branch.

The sixty-second ceiling will not be raised to make this pass.

The engine restart path exists, and a deliberate permanent HOLD blocks it

The worker did not fix this, and was right not to. Its finding, with lines:

engine replacement path, and the owned-engine permanent-HOLD gate that blocks
it, byte-identical at origin/master (launcher.py:5547-5554, 5562-5568,
5651, 5840-5841). Master soak already had the short-profile engine fault
at 185s and the same SIGTERM recovery loop.

Measured, not inferred: _handle_engine_exit is 239 lines at master and 239 at
this head and identical; _check_engine_health 113/113 identical; _start_engine
443/443 identical. This branch did not cause it and does not contain it.

And its own conclusion:

The launcher's owned-engine restart-and-election path remains a master-level
open design item (ROADMAP.md:252 "crash/restart/election evidence"); the
permanent HOLD is deliberate and must not be weakened. Any fix belongs in a
launcher change on master, not this PR.

What this means for the objective, stated plainly

The short profile faults the engine on schedule at 185 seconds and requires
recovery inside a 60-second ceiling. The launcher, by design, holds instead of
restarting.

So a sealed short-profile PASS is not reachable until the launcher can restart
a faulted engine. That is not a defect in the soak and not a defect in this
pull request. It is a disagreement between a deliberate design decision and the
stated objective of a week of continuous running.

The bridge, faulted in the same run, came back:
ZMQ bridge replacement committed; reason=data-flow generation=1 restart_count=2.
The mechanism exists in the system. It is not wired for the engine.

The three ways out, and all three are the owner's

1. The launcher gains an engine restart path. This is ROADMAP.md:252,
   already a planned milestone item, and it restores a role the system already
   owns rather than adding a capability it never claimed.
2. The soak stops faulting the engine. That weakens the evidence, and it is
   not proposed here.
3. The week runs as-is, with it written down that an engine crash ends the
   run and everything after it is lost.

Option 3 is the honest description of today. It should be known before anyone
travels to the stand, which is why it is recorded here rather than left in a
worker's report.

What the soak has now proven it can do

Nine barriers cleared, each named by evidence and each fix verified by reverting
production. At the current head the run starts, passes its exact-six integration
gate, brings up all four roles, writes 38 samples and 4 runtime closures,
injects a scheduled fault, and recovers the bridge from it.

The instrument works. What it found is a real endurance limit.

@codex review

Head is f00cde45e2bcc6a2f06af521cdbcdf8f60164ff2. A lane closed the live review findings; the coordinator ran the landing gates on this tree -- byte-order-mark, encoding and parse checks on every changed file, a refusal on any tree that deletes more than it adds, ruff check and ruff format --check on the changed Python, the derived documentation pair regenerated to a fixed point, and the documentation gate green.

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@codex review

Head under review: f00cde45e2bcc6a2f06af521cdbcdf8f60164ff2

Full check set green with zero pending, and gate.sh measures zero findings at
this head. Requesting the verdict that binds to this exact commit.

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Root cause of the short soak's FAIL, measured

The short soak ends with:

"reason": "source stack did not reach the exact four-role startup cut",
"error_type": "_RunnerFoundationError", "state": "FAIL"

log-launcher.txt in the same evidence directory says why the stack never
started, and it is not a launcher fault:

cryodaq.gui._theme_loader.ThemePackError: theme pack 'warm_stone' is unavailable
RuntimeError: Default theme pack invalid: theme pack 'warm_stone' is unavailable

The launcher dies at IMPORT time. cryodaq/gui/theme.py calls resolve_theme()
at module level, and resolve_theme() raises when the DEFAULT pack file cannot
be found.

Why the pack is missing

It is not missing from the snapshot. config/themes/warm_stone.yaml is tracked
(12 packs are), so git archive carries it into the sealed tree.

It is missing because of where the child is told to look.
_source_environment sets

"CRYODAQ_ROOT": str(resolved),

where resolved is the ISOLATED source root — the directory the same function
creates home, tmp, cache and xdg-config inside, and which its own guard
requires NOT to be under the repository. CRYODAQ_ROOT is the highest-priority
input to get_project_root(), and get_config_dir() is get_project_root() / "config". So the child looks for config/themes/warm_stone.yaml under a
directory that was never given a config/ at all.

Measured, not read

On Ubuntu 22.04.5, in a native worktree at master db55845132:

• A. CRYODAQ_ROOT set to an empty directory, importing cryodaq.gui.theme:
  exit 1, and the traceback is character-for-character the one in the soak's
  log-launcher.txt.
• B. CRYODAQ_ROOT set to the application tree and CRYODAQ_STATE_ROOT set
  to the isolated directory: exit 0, theme warm_stone, config resolves inside
  the application tree, data resolves under the isolated root.

Tool: evidence/tools/rootprobe.sh in the workspace layer.

What this suggests the fix is

Not copying config/ into the isolated root. paths.py already documents the
seam for exactly this case:

CRYODAQ_STATE_ROOT may independently relocate writable data/ and
logs/ state without changing the read-only configuration and TSP roots.
This keeps sealed/exported application trees immutable during verification.

So the two roots carry different things and the child currently sets only one of
them, for the wrong half:

• CRYODAQ_ROOT should be the sealed snapshot — read-only configuration,
  which is what the application needs to start.
• CRYODAQ_STATE_ROOT should be the isolated root — writable data/ and
  logs/, which is what the isolation is actually protecting.

Measurement B is that arrangement, and it starts. The snapshot lives under
/tmp, so the existing guard that the isolated root is not inside the
repository is unaffected.

One thing deliberately NOT proposed here

resolve_theme() refuses to start the program when a colour file is missing.
That refusal is pinned by tests/gui/test_theme_loader.py::test_missing_default_pack_raises,
so it is deliberate, and there is a defensible reading of it: a tree with no
config/ has no safety configuration either, and the theme is merely the first
thing that notices. Changing it is a behaviour decision and is going to the
owner's queue with this measurement attached, not changed here.

What is worth fixing regardless is the message. An operator who reads
Default theme pack invalid will go looking for a theme problem, when the
condition is that the configuration directory is not where the program was told
to look. The message should name the resolved path and the root that produced it.

Follow-up, because the strict check is not wrong everywhere — only where the gate runs

I said the strict _runtime_library_root() check refuses on the hosted runner's conda prefix. That stands. The obvious next question is whether it would also refuse on the machine this pull request exists to serve, so I measured that rather than assuming either way (evidence/tools/library_root_probe.py, which reports each condition separately so the answer is a reason and not a verdict).

On the laboratory machine — Ubuntu 22.04.5, the miniforge environment the soak worktrees actually use:

interpreter: /home/cryodaq/miniforge3/envs/cryodaq-lab/bin/python (3.14.6)
library root: /home/cryodaq/miniforge3/envs/cryodaq-lab/lib
 uid=1000 gid=1000 mode=0755, resolves to itself
 ok is a directory
 ok is NOT a symlink
 ok resolves to itself
 ok owner is root or this user
 ok not group- or world-writable

Every condition passes there. So the design is not unusable in the laboratory — it is unusable on the hosted runner, which is where a required check runs, and that alone blocks this pull request.

That narrows the repair rather than widening it. It is not a case of the check being wrong about what a safe library root is; it is a case of a refusal being the wrong response to a root it does not recognise. The owner's standing rule says exactly that:

keep the check, guide instead, record the receipt

which is the shape #87 already has: resolve the directory, set LD_LIBRARY_PATH when it exists, do not when it does not, and never raise. Keeping the strict conditions as a recorded observation rather than an activation refusal would preserve everything this branch was trying to protect and still run on both machines.

One thing I have not measured and am not guessing at: why the runner's prefix fails the check. The traceback proves it does; which of the five conditions it trips would need the probe run inside a job, and that costs a round.