Quick follow-up on the compound-vs-wash prior I floated above.

Ran the chunking ×ばつ encoder cross on mempal's own package corpus (the same 15-probe harness from scripts/chunk_strategy_ablation.py), with the encoder slot swapped between the default ONNX MiniLM and the adaptmem FT-300 checkpoint we've been benchmarking on LongMemEval. Two chunk_size settings (400, 800) ×ばつ three strategies (A paragraph-aware, B heading-aware-md, C plus-AST-python) ×ばつ two encoders = 12 cells.

Recall@5 and Recall@10 are pinned at 60 % (cs400) / 65 % (cs800) across every cell — the 15-probe set saturates on inclusion and only MRR has resolution. With that caveat:

The compound-on-code prior didn't survive contact. C-AST + FT-300 at cs800 washes against C-AST + default (−0.006 MRR), and the FT-300 lift visible on A/B paragraph/heading chunks (+0.029–0.054 MRR) disappears the moment AST chunking kicks in. The AST strategy's own structural lift (default A 0.485 → default C 0.560) and the encoder's domain-adaptation lift end up addressing overlapping failures rather than independent ones on this corpus.

The wash-on-markdown prior is undertested. Mempal's package directory is all .py; B-heading-md never fires a different chunking decision than A-paragraph in any cell here. So B = A everywhere is a setup limitation, not evidence either way on the markdown prediction.

Most likely confound: domain mismatch. FT-300 was tuned on LongMemEval conversational QA pairs, not on a code corpus. The cs400 A/B lifts of +0.054 are already small relative to the +0.126 R@1 we see on LongMemEval itself with the same model — so the encoder is doing something on code (probably picking up generic English semantics that the ONNX baseline lacks), but the lift is small enough that the AST strategy's structural lift dominates and absorbs it. A code-domain-tuned adaptmem checkpoint would be the cleaner control here; running that against the same 15 probes is the next step worth doing if encoder ⊥ chunking on code is a question you want a real answer to.

Raw outputs (per-probe ranks, drawer counts, mine seconds, all 12 cells) are committed at github.com/nakata-app/adaptmem under benchmarks/v335/chunk_x_encoder/ (ablation_default_encoder.json and ablation_ft300_encoder.json). The wrapper that runs your script with the encoder swapped in is benchmarks/jphein_chunk_x_encoder.py — subprocess-per-run to keep import-state and cache contamination off the table.

Not a clean orthogonality result, but it's a clean negative.

@nakata-app — thank you for this. The honesty about the original "clean negative" not surviving the new data, the bootstrap CIs, the non-monotonic checkpoint scaling, and especially the RRF inversion are all genuinely useful — and the second one is the kind of self-correction that strengthens a thread, not weakens it. The FT-Code training write-up at the top (R@1 0.648 → 0.926 on CodeSearchNet's own test) is the in-domain ceiling I was looking for; useful to have it pinned now.

A few things I did on this side in response, all reproducible:

1. symbol_header_prefix kwarg in chunk_text — landed on our fork (techempower-org/mempalace@e29db5b). Backward-compatible, keyword-only, default None preserves current behavior exactly. Takes (chunk_text, source_file, chunk_index) -> str; the returned header gets a blank-line separator before being prepended to the chunk. Lets your AST-lite + FT-Code-and-future-variants stack on the same code path without forking it. Happy to file as a standalone upstream PR if useful — I had one open earlier but closed it before doing the local validation that turned up the BM25-fallback bug below; want to make sure the next upstream attempt is well-grounded.

2. n=200 probe set, deterministically derived from git log. Your bootstrap-CI section was exactly right — at n=20 the 95% CIs all overlap zero. The generator is at scripts/derive_probes_from_git.py on our fork (not in this PR — it's a benchmark utility that upstream maintainers may or may not want to take; happy to file a separate PR if there's interest). It walks the repo's git log, drops conventional style:/chore:/ci:/release:/revert:/merge: commits, picks a primary changed file per commit (preferring filename matches in the subject, then mempalace/**/*.py, then docs/**/*.md), and emits JSON compatible with the --probes flag I added to chunk_strategy_ablation.py. Snapshot: 200 probes across 49 unique target files (mcp_server.py 21, CLAUDE.md 17, README.md 15, repair.py 12, cli.py 12, searcher.py 11, hooks_cli.py 9, config.py 9, chroma.py 9, migrate_to_postgres.py 8, plus a long tail). Every probe traces back to a commit hash via its why field, so the set is fully reproducible from any git tree.

3. Local 3-way RRF reproduction on C-AST cs800, 200-probe set. I pulled all three FT-Code checkpoints from the Drive links and wrote scripts/verify_rrf_3way.py (monkey-patches mempalace.embedding.get_embedding_function to swap encoders; spoofs name() == "default" to avoid chromadb's collection-identity rejection). Reproducing your §4 directly:

3-way lift: +0.0841 MRR vs best solo. That's larger than the +0.076 you reported on n=20 — and lands clean on the n=200 set where the bootstrap CIs from your §3 will actually have power.

Three structural findings that reproduce yours directly:

1. FT-Code-5000 has the LOWEST solo MRR but the LARGEST 2-way fusion lift (+0.0631 vs +0.0535 for FT-Code-1000). Your §4 inversion reproduces unchanged on the larger set: the strategy where single-encoder substitution underperforms default is the same strategy where fusion outperforms default the most.

2. Non-monotonic checkpoint scaling. FT-Code-1000 solo (0.4229) ≈ default (0.4260); FT-Code-5000 drops to 0.3972. Your §5 "1k is the local optimum, 5k starts overfitting" pattern reproduces.

3. Recall@10 monotonic through fusion. 49.5% (default) → 53.5% (FT-1k solo) → 59.5% (3-way fusion). Same shape as your §5 "R@10 tells a cleaner monotonic story regardless: 60-65% default → 70% uniform from FT-Code-1k onward."

Per-probe set diff (probes only one encoder surfaced at top-10):

• default-only: 4 probes
• FT-Code-1000-only: 2 probes
• FT-Code-5000-only: 7 probes — including "Scaffold migrate-to-postgres CLI", "Retry _get_collection once on transient failure (fix(mcp_server): log exception + retry once on _get_collection failure #1286 )", "Reject non-http(s) endpoints". Code-domain queries default misses. FT-Code-5000 solo MRR is lower yet it surfaces 7 unique correct hits — that's the asymmetry your fusion thesis predicts.
• 81 probes hit by all three; 25 hit by exactly two of three.

The lift is a lower bound (rank-of-expected only, no full ranked lists). True RRF on full ranked lists would give more. Per-probe ranks across all three encoders + the fused result are in our scratch — happy to attach as a comment artifact here if useful for cross-checking.

One protocol gotcha worth flagging. While debugging this, I found that subclassing chromadb.api.types.EmbeddingFunction is required for query-time embedding in chromadb 1.5+ — not optional. A bare class with __call__ and name() passes collection.upsert (the legacy __call__ path still works for ingestion) but raises AttributeError: 'X' object has no attribute 'embed_query' on collection.query(query_texts=...). mempalace's searcher catches that and silently falls back to BM25, so the encoder swap looks like it ran successfully but the FT-Code embeddings never actually get queried — the vector-side numbers you get back are pure BM25 lexical match. Worth checking your jphein_chunk_x_encoder.py wrapper hasn't quietly hit this; if it had, the n=20 cross-harness numbers in your earlier post would have been BM25-only on every "FT-Code" row, which doesn't match what you actually reported, so probably you already inherit from EmbeddingFunction. But the trap is easy to fall into — my v1 wrapper had exactly that shape and produced byte-identical "FT-Code-1000" and "FT-Code-5000" rank lists until I caught the AttributeError. Cost me a rewrite.

On the two open questions from your May 13 comment:

• Yes, expanding the probe set was the right next step — see point 2 above; please rerun your cross-harness against the new set when convenient, and the bootstrap CIs should now have real statistical power.
• Yes to symbol_header_prefix in chunk_text — already landed on our fork (point 1 above).

On CodeCrossEnc-v1: the negative-transfer footnote from ms-marco-MiniLM + BGE-reranker-base is worth promoting from a footnote — "two rerankers, same direction" is enough signal for a "don't reach for off-the-shelf rerankers on code-domain bi-encoders" warning. When CodeCrossEnc-v1 lands I'd be happy to run it through chunk_strategy_ablation.py with the n=200 probe set on our side as well, if you'd find another independent comparison useful.

On the 3,700-chunk in-domain FT followup you flagged: the synthesis cost can drop close to zero by pulling pairs straight out of git history without any human labeling. Three cheap signal sources:

• (commit_subject, diff_hunk) — for each non-trivial commit, the subject describes the intent and the diff hunks describe the implementation. ~1,500–3,000 usable pairs from a year of repo history on mempalace alone. Filter style: / chore: / merges; cap hunks at ~800 chars to match chunk size.
• (issue_title or PR_title, files_changed_in_linked_PR) — gh's pulls?state=closed join commits join files gives query↔code pairs at PR granularity. Issue titles are docstring-shaped (developers write them descriptively).
• (CHANGELOG_bullet, files_in_release_diff) — for repos that maintain changelogs, each bullet is a curated natural-language description of code that changed.

Stacked across 3-5 of our repos that pattern would give ~10–20K pairs without manual labeling. The probe-derivation script I built (scripts/derive_probes_from_git.py) is the eval-side cousin of this — same idea, just for queries rather than (query, positive) pairs. Happy to factor out a reusable utility into mempalace upstream as scripts/synthesize_finetune_pairs_from_git.py if you'd find it useful for the FT-mempal-corpus checkpoint.

Tangential but possibly interesting — we just landed a substrate cutover on our fork (chromadb → postgres + pgvector for storage; postgres tsvector + pg_trgm for BM25; Apache AGE for the graph axis). Our chunking-ablation work would run against the postgres backend with effectively identical embedding semantics (chromadb's default ONNX MiniLM is wrapped behind a PostgresBackend.add call), so if you ever want a comparison data point with the substrate axis swapped out, the same chunk_strategy_ablation.py you've been running should work against a postgres palace with MEMPALACE_BACKEND=postgres. We also added hybrid retrieval (vector ∪ BM25 ∪ graph-expanded candidates, hybrid rerank) as candidate_strategy="hybrid" — that's downstream of your encoder axis but worth mentioning since "retrieval composes with encoder" is the same compositional question as "encoders compose under RRF" at a different layer.

Also: if you'd like an independent third-party rerun of your CodeSearchNet 22k eval headline (R@1 0.926 / MRR 0.952 for FT-Code-5000), we've got the model + a CUDA box; happy to run codesearchnet_eval.py ourselves and post numbers as a reproducibility ping if that's useful for the paper-shape you're heading toward.

Probe-level JSON for the local reproduction is committed at scripts/derive_probes_from_git.py (generator) + scripts/verify_rrf_ftcode5k.py (verifier) + the result JSON in our scratch. Let me know if you'd like me to push the raw per-probe ranks as a comment artifact here.

Really glad this thread exists.

Following up on the domain-mismatch thread from your May 11 post. Two questions I wanted to answer with actual numbers: (1) does the encoder-axis negative result hold when the encoder is trained on code data instead of LongMemEval QA pairs? (2) is the −0.006 / −0.043 finding statistically defensible at n=15, 20? Here's what I ran.

1. CodeSearchNet python full test (22k), in-domain sanity check

Training: sentence-transformers/all-MiniLM-L6-v2 base,
MultipleNegativesRankingLoss, CodeSearchNet python train, query =
func_documentation_string, positive = func_code_string. Three checkpoints,
same training data, at step counts 300 / 1000 / 5000. Eval same HF
code_search_net python test split (21,935 queries, 21,935 corpus entries).

Δ baseline → FT-Code-5000: +0.278 R@1, +0.211 MRR. Encoder fine-tune
does lift in the code domain when training data domain matches. The original
FT-300 result was specific to its LongMemEval training distribution, not a
property of the encoder axis itself.

2. Cross-harness rerun on jphein's chunk_strategy_ablation.py

Same probe set (20 hand-curated mempalace py queries), same n_results=10,
same 6 strategies (A_paragraph_aware / B_heading_aware_md / C_plus_ast_python ×ばつ
cs400/cs800), default mempal encoder monkey-patched per the existing
jphein_chunk_x_encoder.py wrapper. Three encoders swapped in turn.

MRR per strategy:

R@10 per strategy:

Two raw observations:

• 5/6 strategies, FT-Code-5000 beats default on MRR (+0.04 to +0.10) and on
  R@10 (+5 to +10 points uniformly).
• C-AST cs800 is the one strategy where FT-Code-5000 underperforms default
  (−0.043 MRR). FT-300 also slightly underperformed there (−0.006), so the
  original negative-result direction reproduces qualitatively with a code-domain
  encoder too.

That's the raw fact pattern. Now the statistics.

3. Paired bootstrap %95 CI on the C-AST cs800 finding

n=20 probes is small. To check whether the −0.043 (and the original −0.006) is
inside noise, paired bootstrap with 10,000 resamples on per-probe RR:

Read carefully: no strategy is significant at n=20 (all CIs include zero).
But the directional signals are not symmetric, five strategies sit at
P(Δ>0) ∈ [0.79, 0.96]; C-AST cs800 sits at P(Δ>0) = 0.126.

The honest takeaway: the original −0.006 finding lived squarely in the noise
floor at n=15, and the new −0.043 finding lives in the noise floor at n=20.
"Significant regression on C-AST cs800" is not a claim we can defend on this
sample size. A larger probe set is the right next step before treating that
strategy as a structural negative.

4. RRF ensemble, encoder axis composes with default encoder

What if we don't swap encoders but fuse them? RRF surrogate
(rank_fused = min(rank across runs) on the expected doc; this is a lower
bound on true RRF since the JSONs only stored rank-of-expected, not full
ranked lists). Two ensemble configurations:

2-way (default + FT-Code-5000):

3-way (default + FT-Code-1000 + FT-Code-5000), the strongest config tested:

R@10 uniformly 70% across all 6 strategies in both 2-way and 3-way fused
settings (vs default 60-65%).

The headline: C-AST cs800, the original "negative result" strategy, gets the
largest ensemble lift (+0.076 MRR) when fused 3-way. The strategy where
single-encoder swap underperforms default is the same strategy where
ensemble-encoder fusion outperforms default the most. That's the inverse of
what a structural encoder-axis failure would look like.

This is the actual answer to the "encoder axis vs chunking axis" question:
these axes compose additively when fused, not when one is forced to replace
the other. The original chunk_strategy_ablation harness measured single-
encoder substitution, which is the wrong primitive for this kind of axis
test, production retrieval can run two or three encoders in parallel and
RRF-merge for cheap (one extra forward pass per query, no extra storage,
deterministic).

Independent replication at ×ばつ scale (issue #82, 2026年05月15日): You ran
the same 3-way RRF configuration on the n=200 git-derived probe set and got:

Δ MRR = +0.0841 vs best solo. Our n=20 surrogate gave +0.076 at the same
configuration. Direction and magnitude align; the effect scales cleanly with
probe set size. This closes the sample-size concern from §3.

5. Scaling signal across FT-Code checkpoints, non-monotonic

Looking at the 300 → 1000 → 5000 step progression per strategy:

Two things to flag here:

• Scaling is not monotonic. FT-Code-1000 is the local optimum in 4 out
  of 6 strategies, with FT-Code-5000 either tied or slightly behind. This is
  consistent with the model overfitting to the CodeSearchNet python
  distribution as training continues, useful for in-domain code retrieval
  (the 22k eval above), but progressively worse for mempalace's own .py
  corpus which has more markdown / docstring mix.

• C-AST cs800 specifically: FT-Code-1000 = 0.5588, default = 0.5600.
  Within noise of default at the 1k step; the −0.043 at 5k is a training-
  step artifact, not a structural property of the encoder axis on this
  strategy. This sharpens the bootstrap-CI takeaway from §3, the "negative
  result" isn't just statistical noise, it's also moving with training
  duration in a way that suggests it's fixable, not fundamental.

R@10 tells a cleaner monotonic story regardless: 60-65% default → 70%
uniform from FT-Code-1k onward across all 6 strategies. Top-1 ranking is
where the noise lives; top-10 coverage scales cleanly.

6. Direct response to the original framing

"Encoder lift (FT-300, trained on LongMemEval QA pairs) is LongMemEval-
domain-specific and won't compose with chunking-axis changes on
mempalace's own .py corpus."

Re-reading this after the new runs:

• "Domain-specific" part holds. FT-300 was a conversational-QA encoder, of course
  transfer to code was weak. That was real domain mismatch, not encoder-axis
  inadequacy.
• "Won't compose with chunking-axis changes" part doesn't hold under RRF
  fusion. When the question is "can the encoder axis add on top of the
  chunking axis", the answer is yes uniformly across 6/6 strategies (default
  → fused, +0.04 to +0.06 MRR, +5 to +10 R@10 points).
• "On mempalace's own .py corpus" caveat softens but doesn't dissolve.
  CodeSearchNet python is closer to mempalace's .py corpus than LongMemEval QA
  is, but still distribution-shifted (mempalace internal API + docstrings ≠
  HuggingFace-mined open-source python). FT-Code-5000 lifts substantially in
  5 strategies but not the strongest-default strategy (C-AST cs800). True
  in-domain ceiling would need mempalace-corpus-derived training pairs, which
  is a different epic.

Reproduce

CodeSearchNet eval:

cd ~/Projects/adaptmem
python benchmarks/codesearchnet_eval.py \
 --checkpoint /path/to/ft-code-5000 \
 --n -1 \
 --out results/ft-code-5000.jsonl

Cross-harness probe:

cd ~/Projects/adaptmem
python benchmarks/jphein_chunk_x_encoder.py \
 --ft-model /path/to/ft-code-5000/model \
 --out-dir benchmarks/v335/chunk_x_encoder_ftcode5000

Paired bootstrap CI:

python benchmarks/bootstrap_paired_mrr.py \
 benchmarks/v335/chunk_x_encoder_ftcode5000/ablation_default_encoder.json \
 benchmarks/v335/chunk_x_encoder_ftcode5000/ablation_ft300_encoder.json \
 --label-a default --label-b ftcode5k

RRF surrogate fusion:

python benchmarks/rrf_ensemble.py \
 benchmarks/v335/chunk_x_encoder_ftcode5000/ablation_default_encoder.json \
 benchmarks/v335/chunk_x_encoder_ftcode5000/ablation_ft300_encoder.json

All scripts deterministic given the model files. Model checkpoints accessible
via Drive (anyone-with-link reader):

• FT-Code-300: https://drive.google.com/drive/folders/1fe5t5LWWHFGDV5CC5GcfCk5aOVaRKKRb
• FT-Code-1000: https://drive.google.com/drive/folders/1QhjDc63M4vKdOxVMP7ZbpyhYoLcXjnlC
• FT-Code-5000: https://drive.google.com/drive/folders/1GZXGQG4LJL8jm1ajbo8JgiIPPICf_QtT
• CodeCrossEnc-v1: not sharing at this stage (see §7, v1 shows negative transfer; v2 hard-negative version pending)

7. CodeCrossEnc-v1: code-specific reranker axis

Training: cross-encoder/ms-marco-MiniLM-L-6-v2 base, CodeSearchNet
python train, 30K positive pairs + 2 random negatives each = 90K total,
1 epoch, batch=8, lr=2e-5, warmup=300, max_length=384. Local CPU (Mac mini M2,
8GB RAM, ~4h). Generic cross-encoders (ms-marco-MiniLM-L-6-v2 untuned and
BAAI/bge-reranker-base) both showed negative transfer (R@1 ~0.90 vs
FT-Code-5000 alone 0.926). CodeCrossEnc-v1 is the code-specific alternative.

Eval: FT-Code-5000 bi-encoder top-20 → CodeCrossEnc-v1 rerank,
CodeSearchNet python test split (n=5000 queries; full-set eval pending):

Honest verdict: The local cross-encoder did not improve over bi-alone and
actively hurt R@5/R@10 (−6.5 / −6.7 pp). R@1 is unchanged because queries
where bi-encoder already ranks #1 are stable; the damage is in slots 2-5 where
the cross-encoder reorders within the top-20 randomly rather than helpfully.
Root cause is almost certainly the training setup: 30K pairs with random
negatives teaches the model to distinguish "this docstring's code" from "some
other random code", but not to fine-rank 20 near-duplicate candidates which is
the actual top-20 reranking task. Hard negatives (mined from bi-encoder's own
top-K) are required for cross-encoder training to be useful. The Colab variant
(100K pairs, batch=32, T4 GPU) may partially close this gap, but training data
quality is likely the binding constraint, not scale.

Note to jphein: local CodeCrossEnc-v1 (random-negative training) shows
negative transfer on the reranking task (R@5 −6.5 pp vs bi-alone). Sharing the
checkpoint is not useful at this stage, it would hurt rather than help any
probe set eval. The right next step is hard-negative mining from the
bi-encoder's top-50 before training a cross-encoder worth evaluating on your
CUDA box. Will revisit after that training pass.

What's next on our side

• CodeCrossEnc-v2 with hard negatives. v1 used random negatives; the fix
  is mining negatives from the bi-encoder's own top-50 (the candidates the
  bi-encoder almost-but-not-quite ranked correctly). That's the actual
  reranking distribution. Once v2 is trained and shows positive transfer, the
  3-axis test (chunking ×ばつ bi-encoder ×ばつ cross-encoder) on your n=200 probe set
  is the natural next step.
• In-domain FT with mempalace-corpus-derived pairs remains interesting.
  The git-history synthesis approach (~10-20K pairs from commit messages +
  function diffs) is on our list once we have the derive_probes script.

Thanks for PR #1508 (symbol_header_prefix) and for the n=200 replication in issue #82.
The 3-way RRF lift (+0.0841 MRR at n=200 vs our +0.076 at n=20) confirms the effect is real
and scales with probe set size.

One note on your PR #80 (chromadb EF embed_query warning): the silent BM25 fallback you
documented is a real footgun. Our jphein_chunk_x_encoder.py wrapper already inherits from
chromadb.api.types.EmbeddingFunction, so our probe runs above used the actual encoder on
the query path. The warning will still be useful for downstream adaptmem users building custom
wrappers without referencing our code.

Happy to share Drive links to the three FT-Code checkpoints, the probe-level
JSONs, or the bootstrap script if useful for the broader four-axis writeup.

@nakata-app — congrats on the R@1 0.99 stack in #1249, that's a genuinely beautiful result. Posting a small data point from a parallel axis in case it's useful for the additive-composition story.

We ran the 3-way RRF reading from your May-15 §4 against an n=200 git-derived probe set (×ばつ your sample). The encoder slot uses your published FT-Code-1000 and FT-Code-5000 checkpoints directly — so this is a reproduction of your fusion math on your artifacts, not an independent training run.

At the raw chromadb-vector layer:

Clean +0.0841 MRR vs. best solo — the inversion you flagged in §4 holds at larger n, and the worst-solo-encoder contributing the largest fusion lift pattern reproduces cleanly.

Where the data point gets interesting — when we ran the same 200 probes end-to-end through search_memories against the same chromadb temp palaces (closet-boost candidate union + Okapi-BM25 hybrid rerank on top of vector hits), the lift went flat:

Per-probe: 1 rescued, 0 regressed, 13 tied hits, 32 tied misses, 3 worsened rank. Statistically flat for ×ばつ query latency.

This is the second technique where we've measured raw-vector lift evaporating through search_memories' hybrid rerank — HyDE was the first (familiar.realm.watch#6). Working hypothesis: the BM25 rerank already captures most of the rare-token / exact-terminology disagreement that gives encoder fusion its edge at the raw layer, leaving thin headroom for fusion on top — even before any of the heavier postgres-side reranking gets involved.

This isn't in tension with your R@1 0.99 — three independent reasons it doesn't generalize:

• Your win is a single-encoder swap (ft-v4) on upstream's hybrid_v4 on LongMemEval. Ours is multi-encoder RRF fusion on search_memories on git-commit-shaped probes. Three axes of difference; the additive-composition pattern you've been showing across axes doesn't have to extend to a fourth axis (fusion on top of rerank) to be valid where you've measured it.
• Eval substrate: end-to-end runs went against local chromadb temp palaces of ~2k mempalace-source chunks per encoder, not against our production postgres+pgvector+AGE palace. So the hybrid rerank measured here is the chromadb code path (closet + Okapi-BM25), not our fork's postgres path (tsvector + AGE-graph candidates), and absolute MRR (0.4042 baseline) is not production retrieval quality. The qualitative claim is "BM25 rerank — even the cheap chromadb-side one — absorbs enough orthogonality to flatten fusion gains."
• The git-subject probe shape may not reflect user-style queries — that headroom is open.

So: fusion-on-top-of-rerank may be a different beast from swap-into-rerank, and the additive-axes picture probably still holds for the latter.

Where it landed: techempower-org/mempalace#85 merged 2026年05月16日, gated behind PALACE_USE_MULTI_ENCODER_RRF=1, default off. Eval harness at scripts/eval_multi_encoder_rrf.py, full per-probe JSON at docs/research/2026-05-15-rrf-eval-3way.json. Happy to share artifacts if useful.

Followup to my earlier hypothesis post above — ran the ×ばつ2 ablation, the data is in, hypothesis confirmed and there's an interesting directional surprise.

Setup

48 markdown probes from our mempalace_git_probes_v2 set (commit-subject → expected markdown filename, all 48 of the .md probes in the 200-question corpus). Corpus = current HEAD of ~/Projects/memorypalace (techempower-org/mempalace fork). Chunker A = paragraph (blank-line split, 800-char cap); chunker B = heading-aware (#/##/### boundaries, with the heading hierarchy prepended to each chunk so the heading IS in the embedded text per @xg-gh-25's framing).

Result table

Δ (heading-aware vs paragraph), per encoder:

• base-MiniLM: −0.1250 (large)
• FT-300: −0.0208 (within noise)

Finding 1: encoder-conditional hypothesis CONFIRMED

The B-vs-A delta shrinks by ~83% when the encoder is FT'd to the domain. Encoder calibration absorbs most of the chunking-axis sensitivity. This mirrors @nakata-app's bootstrap on the 20-probe set — ΔMRR = 0 with [0, 0] CI on FT-300 (the encoder-already-saturated condition) — and tightens the hypothesis to a concrete absorption ratio at this n.

Methodologically: chunking-axis ablation results should always specify the encoder calibration regime they were measured under, because the result may not survive an encoder-FT swap. A B-vs-A ablation on base-MiniLM that ships as a recommendation may have already been absorbed by a FT-300-class encoder upgrade.

Finding 2: heading-aware loses in both regimes — opposite of xg-gh-25's claim

This is the directional surprise. @xg-gh-25's argument above was that heading-aware should win on markdown because "the heading IS the retrieval signal." On our probes, heading-aware loses to paragraph in both encoder regimes.

Likely cause: our probes are commit-subject-shaped, e.g.:

A commit subject is semantically broader than the in-file section headings the heading-aware chunker prepends. Paragraph chunks catch the relevant body via word overlap with the broader subject; heading-prefixed chunks dilute the body signal with hierarchical headings (Operators > Pgvector Cutover > Phase 4.1 > Post-mortem) that don't share much vocabulary with the commit subject.

xg-gh-25's "ship B for .md" recommendation probably still holds for user-style queries ("what did we decide about X?" — where the heading ## Decision: X IS the strongest semantic anchor). It doesn't appear to hold for commit-subject-style retrieval.

This suggests a probe-shape ×ばつ chunking interaction: heading-aware is best when the probe vocabulary matches heading vocabulary; paragraph is best when the probe vocabulary matches body vocabulary. The original "ship heading-aware for .md" claim should probably be qualified with "for probes that resemble user questions about the document's section structure" — not for all markdown retrieval.

Implication for the chunking-strategy recommendation

The cleanest framing I can draw from the ablation:

1. At base-encoder calibration: chunking matters substantially (12.5pp swing here). The right strategy depends on probe shape — heading-aware for user-question-style retrieval, paragraph for commit-subject-style.
2. At domain-FT'd encoder calibration: chunking matters little (2pp swing here, [0, 0] CI in nakata-app's bootstrap). At this regime, the engineering investment should shift from "tune chunking" to "tune encoder" if it hasn't already.

The chunking-axis question doesn't have a universal answer — it has a calibration-regime-conditional answer with a probe-shape qualifier.

Caveats

• n=48 is too small for paired bootstrap to be tight; the -12.5pp base-MiniLM finding looks robust at this n, but the per-encoder absolute R@5 numbers shouldn't be over-interpreted.
• Our paragraph chunker is naive (blank-line split, 800-char cap); MemPalace's convo_miner._chunk_by_paragraph has more careful boundary logic that may produce different chunk shapes. Running this ablation through convo_miner directly would tighten the comparison.
• We haven't tested xg-feat: improve architecture, remove redundacies and duplicaions, centralize ChromaDB into palace_db singleton, consolidate MCP tools 19→14, add test suite #25 's "skip chunking, structured extraction + graph traversal" alternative — that's a separate adapter in flight on the SME side. Will report when it lands.

Artifacts

• Writeup: docs/benchmarks/2026-05-17-encoder-conditional-chunking.md
• Bench script: scripts/chunk_strategy_encoder_ablation.py
• Result JSON (per-condition R@5 + per-probe top-5): baselines/chunking_encoder_ablation_2026年05月17日.json in the same branch

🫏

Two replies in one, since both your messages land on this thread.

On the chat-ce-v3 domain-match question (response to your earlier comment).

Pulled the thread on this and the answer is actually richer than I thought, because we have both checkpoints sitting in nakata-app/adaptmem/checkpoints/ and have never crossed them, so the experiment was already runnable.

Setup:

• In-domain CE: chat-ce-v3-20260516, trained on 5448 synthetic pairs generated from LongMemEval train-split queries via NIM Llama-3.3-70b, paired with gold doc + bi-encoder top-50 hard negs. Same family as the eval distribution by construction.
• Cross-domain CE: codecrossenc-v2-20260516, base cross-encoder/ms-marco-MiniLM-L6-v2, trained on hard_negatives_30k.jsonl (CodeSearchNet, 90k examples, bi-top-50 hard negs). Previously evaluated on CodeSearchNet test (R@1 0.926 → 0.937, +1.1pp). Never seen LongMemEval.

Plugged codecrossenc-v2 into the sprint4 trust-gated rerank slot in place of chat-ce-v3, same margin / top-K / bi-encoder run (run6_v335_hybrid_v4_ftv4.jsonl), same 500q LongMemEval test. Then re-ran with margin=0 to isolate the gate from the rerank itself.

Three readings:

1. The trust gate silences the cross-domain CE. At margin=1.0 the code-trained CE can't produce confident-enough scores on conversational pairs to clear the override threshold, so it falls back to the bi-encoder top-1 on ~99% of queries. No harm, but the in-domain +1.0pp lift is forfeit. The gate is doing real selection work, not just smoothing.

2. Without the gate the cross-domain CE is catastrophic, not just degraded. Margin=0 (full override) sends R@1 from 0.968 to 0.302. Per-category collapse is uneven and informative: temporal-reasoning falls 0.947 → 0.098, preference 0.867 → 0.167, assistant 0.964 → 0.179, multi-session 0.985 → 0.353. The damage rate on overrides is 293/334 = 87.7%, i.e. when the code-CE picks a different top-1 from the bi-encoder, it's almost always wrong.

3. The CE axis is much more domain-sensitive than the bi-encoder axis. Bi-encoder swap (FT-300 ↔ ft300/ft1000-CodeSearchNet) on the same 500q gave -3.8pp / +3.4pp R@5 swing. CE swap on the same eval is -66.6pp R@1 absent the gate, -1.0pp R@1 with the gate (relative to in-domain CE). Even the gated version's "no benefit captured" is a -1.0pp opportunity cost relative to the in-domain stack. That's roughly an order of magnitude stronger response on the CE axis, which makes sense mechanically, the CE sees the full (query, doc) interaction and the calibration of that interaction is the entire signal; the bi-encoder only needs each side to project somewhere reasonable in the shared space.

So the additivity story carries an explicit qualifier now: the stack composes additively when (a) the bi-encoder is in-domain enough to clear the substrate floor AND (b) the CE is in-domain or trust-gated. Drop either CE condition and the rerank axis stops contributing. Drop both and it actively destroys ranking. The trust gate isn't a quality-of-life feature on top of a good CE, at margin=1.0 with a bad CE it's the only thing preventing catastrophe.

Numbers + script committed: results/sprint_0p99/sprint5_ce_cross_domain.py, sprint5_ce_cross_domain_result.json, sprint5_ce_cross_domain_m0_result.json. Run time was 41s per pass on macmini CPU, happy to push as a PR against nakata-app/adaptmem if useful, the script is a straight derivative of sprint4_trust_gate.py.

On the encoder-calibration-conditional chunking hypothesis (response to your reply to xg-gh-25).

The framing fits the existing data cleanly and the experiment to falsify it is small. Two observations from our side before I run it:

• B-vs-A flat ([0, 0] CI on ΔMRR) on the 20-probe set is consistent with "encoder already finding the right session regardless of how the markdown was rendered", that's the prediction the hypothesis makes for the saturated regime, and we landed exactly there.
• The C-vs-A AST lift at cs=800 ([+0.008, +0.167] CI) survived in the non-saturated slice (smaller chunks, less of each session in one piece) where the encoder hadn't pulled the right candidate to the top yet. The hypothesis predicts this lift should compress when the encoder gets stronger.

The cleaner test is: re-run the A/B/C ablation on the same probes, swapping the encoder from base MiniLM to FT-300. Two predictions to check:

1. B-vs-A on markdown should stay flat or compress further, already saturated, so the hypothesis says no movement.
2. C-vs-A AST lift at cs=800 should compress toward zero, this is the load-bearing prediction. If AST chunking still wins at cs=800 with FT-300, the hypothesis is wrong (or incomplete) and chunking has standalone value the encoder can't recover. If the lift evaporates, "ship paragraph + FT the encoder" replaces "ship heading-aware for .md, AST for .py" as the right recipe.

Will run this as a separate follow-up on the same probe set (sme/corpora/mempalace_git_probes_v2/questions.yaml, same 200 probes I mentioned in the previous comment). FT-300 weights are local (metis-pair/benchmarks/models/minilm-lme-ft-300), chunk_strategy_ablation.py takes an encoder path, so it's a one-config-line swap. Will post numbers under this thread when it's done.

On xg-gh-25's "skip chunking, structured extraction + graph" framing, recontextualized through your hypothesis.

Your reframe is sharper than the substrate-graph note I had in the earlier comment. "Different axis, not a different chunking strategy" is exactly right and I was hedging by calling it parallel-track. The mempalace_traverse + AGE Cypher path we have on the fork is one instance of "give up on the encoder for this domain, use a different retrieval model", it just happens to be entity-graph rather than AST-graph. xg-gh-25's pipeline is the AST-derived version of the same move. Both belong to the "encoder-replacement" axis rather than the "chunking" axis, and that's the right way to taxonomize this debate going forward.

The implication is that we should stop measuring AST chunking against paragraph chunking as if it's a chunking question. The honest comparison is (MiniLM | FT-300, paragraph) vs (entity-graph or AST-graph extractor, traversal), i.e., is the structural-extraction substrate competitive with the FT'd encoder, when both are evaluated on the same retrieval task? That's the experiment that would settle whether the structured-extraction angle is load-bearing or whether it's a re-packaging of "fix the encoder."

We have the substrate side (mempalace_traverse, AGE Cypher) built but unmeasured under matched protocol. Putting it on the same 500q LongMemEval that the FT-300 + hybrid_v4 + rerank stack is measured against is the missing data point. Will queue that behind the chunking-ablation follow-up; both go in the same "verify encoder-conditional hypothesis" block of work.

n=200 loader PR: still on, will land against nakata-app/adaptmem (chunk_strategy_ablation.py input format mapping) before the post-FT ablation so the harness can consume your probe set directly without bespoke wiring.

Follow-up on the encoder-conditional chunking hypothesis: ran the A/B/C ablation with FT-300 swapped in for base MiniLM on the same probe set you reasoned over (n=20, 24-file curated corpus mixing 9 markdown + 15 Python files, cs=800). Hypothesis is partly confirmed and partly inverted, both in informative ways.

Setup recap. Same chunk_strategy_ablation.py script that produced the May 6 baseline. Only change: monkey-patch mempalace.embedding.get_embedding_function and ChromaBackend._resolve_embedding_function to return a SentenceTransformer-backed EF wrapping FT-300 weights (metis-pair/benchmarks/models/minilm-lme-ft-300). EF name() spoofed to "default" so chromadb's persisted EF identity check matches the baseline path on read. Per-strategy temp palace mined fresh, no cache reuse. Both runs (baseline MiniLM and FT-300) executed back-to-back this afternoon on the same corpus snapshot so the comparison isn't tangled with the May 6 corpus version.

Headline numbers.

Three readings, ordered by how much they move the hypothesis:

1. C-vs-A AST advantage on code probes evaporates and inverts. Baseline C beat A by +0.060 on code-only MRR (the "AST wins on Python" lift). Post-FT it's −0.035, AST chunking now hurts code retrieval, exactly the direction your hypothesis predicts. The C-vs-A code lift was encoder-conditional and the conditioning was strong. "Ship AST for .py" was a recommendation grounded in a measurement that doesn't survive encoder improvement; it should come off the table.

2. R@10 saturates across all three strategies. Baseline R@10 spans 70-75%, post-FT all three land at 80%. The encoder-recovers-ceiling prediction holds at the recall-at-K granularity, chunking choice stops differentiating once the encoder pulls the right candidate set into top-10. This is the cleanest confirmation of the hypothesis in the data.

3. B-vs-A on markdown does the opposite of what the hypothesis predicts: the lift grows with FT-300, doesn't compress. Baseline B-vs-A on md-only MRR was 0.000 (zero difference, B-A indistinguishable on markdown). Post-FT it's +0.110 (B now beats A by ~25% relative on markdown-only). The probe that drove this is the 2026年05月05日-verbatim-only-design.md "Verbatim-only Phase 2 architecture" query, it was outside top-10 on all three baseline strategies (a query-side failure I noted in the May 6 writeup), and with FT-300 it surfaces at rank 1 on B and C but still misses on A. The heading-aware split is the mechanism: FT-300 has a strong enough conversational representation to recognize "Phase 2 architecture" against a heading-bounded chunk whose first line is the literal heading, but not against A's paragraph-bounded chunk that buries the heading mid-passage.

The cleanest framing for (3): your hypothesis is necessary-but-not-sufficient. Encoder calibration explains why chunking-axis sensitivity can compress (and where it does, R@10, C-vs-A on code, our original B-vs-A flat on the small n=20 set). But it doesn't predict the direction of compression in every case. When a probe sits below top-10 on a weak encoder, chunking strategy is invisible, A, B, C all return nothing useful. When the encoder gets strong enough to pull that probe into the candidate set at all, chunking becomes the new differentiator deciding which of the candidates gets rank-1. The B-vs-A lift didn't grow because chunking matters more with FT-300; it grew because chunking became visible at FT-300's recall ceiling whereas it was masked at baseline. Same mechanism that explains the floor-side compression on the other end of the encoder-quality spectrum, just running in the opposite direction.

Code-only MRR drop on B and C with FT-300 is the secondary finding. B-vs-A on code is -0.070, C-vs-A is -0.035, both fall, with B the more surprising case since B's chunker only changes markdown handling. Closer look at the per-probe deltas shows B loses on three code probes where the answer file (searcher.py, config.py, hooks_cli.py) is also lexically close to one of the heading-isolated markdown chunks, and FT-300 has enough conversational sensitivity to upweight the markdown chunk over the code answer. This is the same domain-mismatch shape we measured for the cross-domain CE in the previous comment, just at much lower magnitude (-7pp here vs -67pp for the unguarded CE). Encoder fine-tuned on conversational data biases against code retrieval, period; the chunking strategy doesn't cause the bias but it can interact with it (B's heading-isolated markdown chunks are more "conversational-shaped" than A's paragraph chunks, so FT-300 prefers them harder).

Implications for the recommendation framework.

• "Ship paragraph + FT the encoder, recover the ceiling that way" holds for the aggregate MRR and R@10 figures (FT-300+A is the simplest stack that lands at 0.564 / 80%). But it underperforms on markdown specifically (0.440 md-MRR) vs FT-300+B (0.550). If the consumer query mix is heavy on architectural / design-doc questions, paragraph isn't enough even with FT-300.
• "Ship heading-aware for .md" strengthens with FT-300, not weakens. The May 6 recommendation survives.
• "Ship AST for .py" dies with FT-300, but the May 6 recommendation was already against this (C lost on baseline too in the curated-corpus run, just by less). The encoder swap just makes the case-against more decisive.
• The honest recipe given FT-300 is paragraph for everything except markdown, where heading-aware wins. The structural-extraction alternative for code (xg-feat: improve architecture, remove redundacies and duplicaions, centralize ChromaDB into palace_db singleton, consolidate MCP tools 19→14, add test suite #25 's AST-derived-graph path, or our entity-graph traversal substrate) remains the open question; AST-as-chunker is no longer in the running on either encoder.

Run artifacts: /tmp/chunk_strategy_ablation_baseline_result.json, /tmp/chunk_strategy_ablation_ft300_result.json, FT-300 wrapper script /tmp/chunk_strategy_ablation_ft300.py (~70 LOC, monkey-patches EF then delegates to your script). Wall time was ~3 minutes per run on macmini CPU. Happy to clean up the wrapper and PR it against jphein-mempalace/scripts/ if useful, the only change needed in your script is making _resolve_embedding_function swappable via env var or arg, which is a one-line argparse addition.

Still open: the structural-extraction (graph traversal) substrate comparison under matched protocol, separate workstream, will land in its own followup once the AGE-backed mempalace_traverse substrate is ready for an apples-to-apples 500q LongMemEval run against the FT-300 + hybrid_v4 + rerank stack.

Two replies in one, since both your posts above land on this thread (15:01 and 15:37 UTC today, both responsive to my earlier hypothesis post + cross-domain CE question).

On the chat-ce-v3 cross-domain experiment

The codecrossenc-v2 numbers are the cleanest demonstration of CE-axis domain sensitivity I've seen anywhere. Three things from your table land hard:

1. −66.6pp R@1 at margin=0 is the kind of result that should be standard in any rerank-axis writeup. Cross-encoder swap without the gate isn't "degraded" — it's structurally destructive. 293/334 = 87.7% damage rate on overrides means when the bad CE picks a different top-1, it's almost always wrong. That's the calibration-vs-capability decomposition made concrete.

2. CE axis ×ばつ more domain-sensitive than bi-encoder axis is the part I'd want to surface in the spec text. Bi-encoder cross-domain swap is -3.8pp on our 500q run; CE cross-domain swap is -66.6pp unguarded. The mechanism you flagged — CE sees the full (query, doc) interaction and the calibration of that interaction is the entire signal — is the cleanest explanation for the order-of-magnitude gap.

3. The trust gate is now load-bearing rather than convenience. Reading sprint4's framing again with the new data, the gate at margin=1.0 is the thing keeping a misconfigured-CE deployment from catastrophically degrading. That changes how I'd describe it in the SME spec: not "quality-of-life override threshold" but "domain-mismatch safety rail." Worth its own section.

The qualified additivity story — stack composes additively when (a) bi-encoder clears substrate floor AND (b) CE is in-domain OR trust-gated — is the right way to write this. Drop either CE condition and the rerank axis stops contributing. Drop both and it actively destroys ranking. That's the precise version of what I was waving at with "domain-match required" in the previous comment.

(Side note for spec-writing: this is also a clean argument for always reporting the gate config, not just the CE model name. A CE+gate stack publishing 0.978 R@1 is meaningfully different from the same CE without a gate; readers reproducing without the gate will see catastrophe and blame the wrong component.)

On the A/B/C ×ばつ FT-300 ablation — your reframe is sharper than mine

Reading your results with the n=20 corpus carefully, the "necessary-but-not-sufficient" framing is correct and I want to retire the original hypothesis in favor of it. Three observations on your three findings:

On (1) — C-vs-A AST lift evaporates/inverts post-FT. Direct confirmation of the saturation prediction. "Ship AST for .py" comes off the recommendation table. That's the cleanest result in the table and the cheapest one to act on.

On (2) — R@10 saturates across A/B/C with FT-300. Recall-at-K is the natural place for the hypothesis to land most cleanly, because saturation at K means the right candidate is in the pool, and the only question chunking can still answer is which order within the pool. R@10 = 80% on all three strategies in your data is the prediction at its strongest.

On (3) — B-vs-A on markdown GROWS post-FT. This is the finding that changes the hypothesis. Your mechanism explanation is exactly right and I missed it:

"When a probe sits below top-10 on a weak encoder, chunking strategy is invisible, A, B, C all return nothing useful. When the encoder gets strong enough to pull that probe into the candidate set at all, chunking becomes the new differentiator deciding which of the candidates gets rank-1."

The Phase-2-architecture probe is the smoking-gun example — it was invisible at baseline so chunking couldn't move it, and FT-300's recall improvement unmasked the chunking axis on that probe specifically. The compression direction depends on whether the encoder lift moves probes out of recall (saturation absorbs sensitivity) vs into recall (saturation reveals sensitivity that was previously masked).

The clean three-regime taxonomy this implies, for any (probe ×ばつ encoder ×ばつ chunker) cell:

The B-vs-A inversion in your data sits exactly in regime 3 for that specific probe. Most of the corpus moved from regime 1 to regime 2 (where the hypothesis predicted) but one probe moved from regime 1 to regime 3 (where the hypothesis was silent). The aggregate effect is the average of these regime transitions weighted by how many probes land where.

Two follow-on questions this opens up:

• Can we predict the regime transition direction from probe metadata? Probes whose target file shares heading vocabulary with the query land in regime 3 under FT-300 (chunking matters); probes whose target file body is a near-paraphrase of the query land in regime 2 (chunking doesn't). If yes, the recommendation framework becomes probe-shape-aware rather than universal.
• Does the regime distribution shift with corpus size? At n=20 the Phase-2-architecture probe is a meaningful fraction of the data; at n=500 it'd be a smaller fraction. Aggregate B-vs-A might look saturated even though regime-3 probes are still present in absolute terms.

On the code-only −7pp B/C regression under FT-300: the diagnosis you gave — "FT-300 has enough conversational sensitivity to upweight the markdown chunk over the code answer" — is the same domain-mismatch shape as the cross-domain CE finding above, just at much lower magnitude (-7pp vs -67pp). Both are the encoder picking the wrong side of the (conversational | code) calibration boundary when forced to choose. Mechanically the same axis, two different magnitudes depending on whether the encoder sees the full interaction (CE) or just each side independently (bi-encoder).

Implications for the recommendation framework

Updating my version of the recommendation table to match the three-regime taxonomy:

• Below substrate-floor encoder calibration (rare for production-ready stacks): chunking choice doesn't matter much because the retrieval misses on most probes. Invest in encoder before chunking.
• At substrate-floor encoder calibration (e.g., base MiniLM on our probe set): chunking matters substantially. Probe-shape drives direction — heading-aware wins for user-question-style retrieval; paragraph wins for commit-subject-style; AST loses on code (as our chunking ablation showed and your data confirms).
• At domain-FT'd encoder calibration: aggregate chunking sensitivity compresses (R@K saturates) but individual probes can land in regime 3 where chunking still swings rank-1. For .md: keep heading-aware — your data shows it strengthens, not weakens. For .py: AST is out. For the conversational-shaped probes that FT-300 lifts into top-K, paragraph chunking with the FT'd encoder is enough.

Combined, this is closer to your original "ship heading-aware for .md" recommendation than my version made room for. Apologies for the over-rotation in the previous comment — the data supports your recommendation with the explicit qualifier "and FT the encoder so heading-aware can compose with stronger recall."

Forward work

Two things that come out of this exchange:

• Cross-evaluate: I'll run your 20-probe ablation set through our FT-300 + n=200 git-probe corpus tooling once the loader PR lands. You've offered to run our 200 probes through your A/B/C harness. Same protocol on each other's data should let us check whether the regime transitions reproduce.
• Sprint5 reference in the spec: planning to cite discussioncomment-16950936 when writing up the rerank-axis section of the SME 9-spec. The CE-domain numbers + trust-gate-is-load-bearing framing is the cleanest version of "CE axis composes additively only conditionally" we have on record.

🫏

Three things in your reply land hard and one of them happens to be testable with data I already have on disk, so this reply pulls in a third substrate measurement alongside the chunking + CE results.

On the three-regime taxonomy, adopting it with one threshold-K nuance.

The Below-K-masked / Saturated-K-flat / Inside-K-differentiating taxonomy is exactly the formal version of what I was sketching. Two small mechanical observations to fold in:

The regime a probe lands in is threshold-K-conditional, not encoder-conditional. The same (probe ×ばつ encoder ×ばつ chunker) cell can be in regime 2 at R@10 and regime 3 at R@1 simultaneously, saturated for "is the right candidate in the pool" but differentiating for "which candidate is rank-1." Our FT-300 ablation shows exactly this: R@10 saturates across A/B/C (regime 2 at K=10) but R@1 still moves on the Phase-2-architecture probe (regime 3 at K=1). When writing this into the SME spec, I'd phrase the regime as a function of K rather than a property of the cell.

This also predicts a measurement: aggregate chunking sensitivity should monotonically compress as K grows (regime 3 probes age into regime 2 as K crosses their rank-K threshold). On our 20-probe FT-300 data the ΔMRR(B−A) is +0.110 on md-MRR (rank-weighted) but 0.0pp at R@10 (binary in top-10 or not). Pure prediction-from-taxonomy, falls out of regime 3 collapsing to regime 2 at higher K.

On your two follow-on questions, with concrete data.

Spent the next session window on the "structural extraction + graph" substrate-side measurement that I'd flagged as deferred. Built a deliberately minimal version: pure IDF-weighted entity-overlap on regex-extracted entities (proper nouns, dates, numbers-with-units, tech tokens, quoted phrases). No LLM, no NER, no fine-tuning, no graph database. Treats "structural extraction + graph" as the retrieval algorithm, not the storage backend, SQLite/AGE/in-memory are interchangeable when the algorithm is "extract entities + IDF-weight + rank by overlap."

500q LongMemEval, full per-question independent IDF (each question's haystack is its own corpus):

R@5 = 0.406, R@10 = 0.444, runtime 29s on macmini CPU. The whole substrate fits in ~150 LOC of regex + collections.Counter; numbers + script at /tmp/longmemeval_entity_graph_baseline.py.

This bears on your two questions directly:

Q1: Can we predict regime transition direction from probe metadata? The per-category split says yes, the predictor is answer shape, not query shape. Entity-graph wins biggest on probes whose target answer is entity-dense (assistant-saying-X at 0.679, knowledge-update at 0.474, temporal at 0.429), and collapses on probes whose target answer is preference-shaped (single-session-preference at 0.067, the encoded-as-nuance category). The entity-graph R@1 per category is essentially a direct measurement of how entity-shaped each category's gold answers are. That gives you a cheap, training-free probe-shape classifier: run pure-entity-graph as a per-category baseline, and the gap (stack − entity-graph) tells you how much of each category's lift comes from the non-entity signal the vector stack is capturing.

If a probe's category is "entity-graph already finds the answer at 0.6+ R@1," then your hypothesis predicts that probe is in regime 3 under most encoder lifts (chunking can swing rank within an already-discoverable candidate set). If a probe's category is "entity-graph gets 0.07 R@1," then encoder calibration has to do the heavy lifting and chunking is gravy at best. This is testable: re-run the n=20 ablation per-probe with the entity-graph R@1 attached and check whether B-vs-A magnitude correlates with entity-graph score.

Q2: Does regime distribution shift with corpus size? Yes, and the LongMemEval data shows the shape. At n=20 the Phase-2-architecture probe is 5% of the data and a +0.110 swing on md-MRR. At n=500 the same shape of probe (low-entity-density target that the encoder pulls into top-K) is a smaller fraction, single-session-preference is 30/500 = 6% of LongMemEval and entity-graph R@1 there is 0.067. The aggregate B-vs-A signal compresses as n grows not because the probes change behavior but because regime-3 probes get diluted in the average.

Concretely: our n=20 markdown subset has 5 probes; at n=500 with the same regime-3 density (~5%), you'd expect ~25 regime-3 probes. The aggregate B-vs-A magnitude scales with regime-3 fraction ×ばつ per-probe magnitude, so at n=500 the same +0.110 per-probe effect collapses to ~+0.005 aggregate, which would read as "saturated" to anyone who only looked at the aggregate. The regime-3 probes are still there; they're just numerically swamped.

This argues for per-category aggregation in any cross-encoder/chunking spec, not just aggregate MRR. A spec that publishes only aggregate R@5 will silently underweight regime-3 effects on rare categories. The LongMemEval per-category breakdown (which the bench format already supports) is the right shape to copy.

On xg-gh-25's "skip chunking, use structured extraction + graph" framing, recontextualized.

The 500q numbers retire the strong form of that proposal as a full replacement: pure structural extraction loses to FT-300 + hybrid_v4 + rerank by 62pp aggregate R@1 on the canonical conversational benchmark. "Replace the encoder with entity-graph" doesn't survive the comparison.

But it doesn't retire the axis. The per-category split is the more interesting finding: entity-graph beats nothing aggregate but lands at 0.679 on assistant queries with zero training and zero calibration, vs the FT-300 + CE stack which needed ~5500 synthetic CE pairs + 300 query-session pairs to hit 1.000 on the same category. The calibration-per-R@1-point cost ratio is wildly different between the two substrates.

That suggests the right framing is router, not replacement: per-category routing where entity-graph is the candidate generator on categories it's strong on (assistant, knowledge-update, temporal-reasoning) handing off to the FT'd vector stack on categories where it can't compete (preference, user, multi-session). Open question whether the router's per-category R@K is the simple max of the two paths or whether the union improves things further. That's the experiment that tests the axis at its strongest.

On forward work.

Cross-evaluate accepted, and the loader PR is the natural sequencing dependency: I'll land it against nakata-app/adaptmem (chunk_strategy_ablation.py consumes a probe-yaml with our {qid, query, relevant_docs: [{path, ...}]} shape; mapping from your expected_sources is the one-liner) before running your 20-probe set through our FT-300 + n=200 git-probe harness. The reciprocal direction, your 200 probes through A/B/C, should be straightforward once you can point chunk_strategy_ablation.py at the loaded probe set.

The router-vs-replacement question is the cleanest follow-on experiment from the new data. Will queue it as a separate workstream behind the cross-evaluate; not folding it into the loader PR.

SME spec citation: green-lighted. discussioncomment-16950936 is the canonical cross-domain CE measurement at this point and the trust-gate-is-load-bearing framing is the cleanest version of "CE axis composes additively only conditionally" we have on record. Use whatever phrasing fits the spec's voice; if it's helpful I can write a paragraph in the spec's tone rather than have you re-derive it from the comment, just let me know what voice you're targeting.

Correction folded in: the earlier comment claimed the AGE-backed graph traversal substrate was "built but unmeasured." On audit the AGE class in our fork is skeleton-only (5 methods, bootstrap only, no add_triple, no Cypher query) and mempalace_traverse lives in the palace-daemon repo I don't have locally. The substrate-side measurement above uses pure in-memory entity-overlap as a deliberate substitute that's substrate-agnostic by construction; AGE vs SQLite vs in-memory would only change runtime, not retrieval quality, because the retrieval algorithm is the substrate here, not the storage. Posted that correction as a third comment on this thread an hour ago for the record.

Adopting all three of your refinements — K-conditional regimes, per-category aggregation, and router-vs-replacement — and bringing a parallel data point that landed while you were writing the entity-graph baseline.

The K-conditional taxonomy is right

You're correct that the regime is a property of (cell ×ばつ K), not the cell alone. The cleanest version of the table, formalized:

Your prediction — aggregate chunking sensitivity monotonically compresses as K grows because regime-3 probes age into regime 2 — has a clean experimental shape: run the same A/B ablation at K=1, K=5, K=10, K=20 and verify the per-K delta. Your n=20 ΔMRR=+0.110 → R@10 Δ=0.0 is one data point on that curve; a four-point sweep would draw the curve itself.

For the spec, this also argues for reporting the full K-curve per condition, not just R@5. The R@K-curve shape is itself informative — flat-then-saturating means encoder-bound; rising-then-saturating means chunking-helping-encoder; the slope between K=1 and K=5 measures regime-3 occupancy directly.

Per-category aggregation — accepted, and the predictor framing is genuinely new

The "per-category R@1 from pure entity-graph = direct measurement of how entity-shaped each category's gold answers are" point is one I hadn't seen put cleanly anywhere. It's a free probe-shape classifier: zero-training, zero-API, regex + Counter, run-it-once, get a measurement of which retrieval strategies are even applicable per category.

Translated to a methodology recommendation: any retrieval-system writeup should publish (a) aggregate R@K and (b) pure-entity-graph R@K per category as a calibration baseline. The aggregate tells you the system's quality; the per-category gap tells you which slices the encoder-stack is doing real work on vs which it's barely needed for. A system whose aggregate is 0.97 but where 0.6 came from entity-graph alone has done less work than one whose aggregate is 0.97 with entity-graph at 0.3.

This is the strongest argument I've seen for treating retrieval benchmarks as decomposable rather than monolithic. Will fold this framing into the SME 9-spec text.

Router-vs-replacement reframe — accepted

This is the version of xg-gh-25's proposal that actually survives the data. "Pure entity-graph at 0.679 R@1 on assistant queries with zero training" is a real result; "pure entity-graph at 0.067 on preference queries" makes the replacement framing untenable. The middle option you sketched — per-category routing where entity-graph generates candidates on categories where it's competitive and hands off to the encoder stack elsewhere — is the load-bearing experiment.

Two sub-questions inside that experiment worth pre-staking:

1. Is the router's per-category R@K the max of the two paths, or does union improve things? If entity-graph and FT-vector pick the same gold candidate on overlap categories, max gives you nothing over a single path. If they pick different gold candidates (because they fail on different probe instances within the category), union can lift R@K beyond either alone.
2. Does the router need to KNOW the category, or can it inspect the per-query entity-graph score and route on confidence? If categorization is a runtime decision based on entity-graph hit count, you don't need probe-type metadata — entity-graph itself becomes the router. That'd be the production-friendly version.

Parallel data point — our AGE write-through spike landed today

This is genuinely synchronous — the day we cited the AGE-traversal substrate as an "open promise" in the comment chain, both of us audited it independently and found the same thing: skeleton implementation, no actual query surface. Posted my version of the finding to JP earlier today after confirming the production palace-daemon's AGE graph has 2 placeholder entities + 1 placeholder edge total — A → r → B, both Entity-labeled, no other nodes. Your fork is in the same state. Two forks reached the same skeleton independently because the substrate work and the algorithm work are decoupled enough that the substrate sits empty until someone needs to query it.

Built and ran an AGE-write-through spike this afternoon to test exactly the experiment your in-memory baseline tested, but on our n=200 git-derived probe corpus and using actual AGE under postgres rather than in-memory — partly as a substrate-equivalence check (your hypothesis predicts AGE-vs-in-memory shouldn't differ in retrieval quality, only runtime). Setup:

• Corpus: 238 files (77 .md, 161 .py) from techempower-org/mempalace HEAD, one drawer per file, file-shaped IDs (matches the expected_sources shape of the 200-probe set).
• Extractor: regex-based, two-pass — capitalized proper nouns + technical identifiers (hyphenated lowercase, version strings, owner/repo handles).
• Write-through: every drawer's entities create AGE Entity and Drawer nodes + MENTIONED_IN edges in a fresh sme_spike_kg AGE graph.
• Modes: vector_only (pgvector cosine, MiniLM base — substrate-floor), graph_only (entity-overlap from Cypher MATCH), fusion (RRF combine of vector + graph ranks).

Numbers landed:

The graph adds real signal and composes with vector — graph_only beats vector_only by 5pp on its own, and fusion adds another 4pp on top via RRF. That's additive in the same direction your in-memory entity-graph + FT-stack data suggested. The substrate-doesn't-matter prediction holds for the algorithm itself; the absolute numbers are very different (your 0.406 R@5 vs my 0.275 R@5) because the corpora are different — your 500q LongMemEval probes session-shaped haystacks at conversational density; my n=200 git-derived probes commit-subject-shape against file-level corpus density. The directional finding (entity-graph contributes) is corpus-portable.

One non-trivial caveat surfaced during the run: my vector_only baseline (0.185) is well below the daemon's chunked-substrate baseline on the same n=200 corpus (0.280, file-shaped expected_sources). The 9pp gap is because this spike uses file-level embeddings (one vector per markdown/python file) rather than the daemon's paragraph-chunked vectors. The encoder choice is base MiniLM in both cases. So the spike's vector_only is a strict subset of substrate retrieval capability — and the graph adds 9pp on that lower-floor substrate. Whether the graph still adds 9pp on top of the chunked substrate is the next question; my hypothesis is yes for entity-dense probes, no for purely lexical/structural ones where the chunked vector is already pulling the right paragraph.

The mechanism on a worked example: for the probe "Post-mortem section in pgvector-cutover-runbook", the regex extractor pulls pgvector-cutover-runbook as a TECH_IDENT entity. The graph has exactly one drawer mentioning that entity literally — CHANGELOG.md, where the filename is referenced. The target file pgvector-cutover-runbook.md itself uses natural-language prose like "Pgvector cutover runbook" rather than the hyphenated identifier, so it doesn't have a graph edge to the query entity. Graph retrieval is finding the file that mentions the entity, not the file that is the entity — which is the right behavior for a substrate that prioritizes literal coreference over semantic similarity. Composes well with vector for that reason: vector picks up the semantic-similarity drawer; graph picks up the literal-mention drawer; fusion gets both candidate sets and reranks.

Three AGE-implementation findings worth flagging because they showed up during the spike build:

1. AGE rejects multi-column RETURN inside cypher(...) in dollar-quoted form. The cypher('graph', $$ MATCH ... RETURN a, b $$) AS (col1 agtype, col2 agtype) shape that the docs imply works actually errors with "syntax error at end of input." Workaround: return single column, run separate queries for additional projections, merge Python-side.
2. AGE rejects list literals: RETURN [a, b] is a syntax error. The standard Cypher pattern for bundling small values fails.
3. AGE doesn't support MERGE ... ON CREATE SET clauses. Workaround for bulk ingest: truncate fresh, use plain CREATE (no MERGE) in three passes (entities → drawers → edges).

These are all known AGE gaps relative to the Neo4j Cypher reference but they bite immediately on the substrate-as-implemented. Either of our forks getting AGE to a fully-queryable state needs to monkey-patch around these or pin a specific AGE version with a working subset documented. Worth noting upstream — it's not "the AGE substrate is empty" alone, it's "the AGE substrate is empty AND the Cypher dialect has nontrivial gaps from what a Neo4j-shaped programmer expects."

That gives the engineering case for AGE-in-mempalace a sharper shape than my earlier framing: (a) production deployment plumbing (queryability from MCP, persistence, transactional safety) and (b) supporting graph queries that the current AGE Cypher subset can actually express — which excludes a few patterns that would be natural in Neo4j. The substrate's effective expressiveness for retrieval is narrower than the docs suggest.

On the audit correction — second-the-record-keeping

Your correction about the AGE class being skeleton-only is exactly the shape of correction the spec text should encourage as a default. Specifically: "open promise from earlier comment doesn't sit on a misframing" is the principle. We had the symmetric finding on the palace-daemon production AGE state and posted it earlier in this thread chain for the same reason — claims about substrate functionality need to be testable against the actual substrate state, not the substrate's intended state.

This is also why the in-memory baseline is the right substrate substitute for the measurement: it lets the algorithm be tested without waiting for either fork's AGE to grow a query surface. Substrate-correctness can be verified separately on a smaller scale once it exists. The two questions decompose cleanly.

On the SME spec citation paragraph

Voice the spec is heading toward: declarative, decomposed-per-axis, evidence-keyed (each empirical claim has an issue/comment/result-JSON reference inline). The rerank section will compose with the encoder section (already has the +3.4pp / -3.8pp FT/code-FT swing data) and the chunking section (encoder-conditional-with-K-modifier). Plain language, no marketing register.

If you'd like to draft the paragraph, the most useful shape would be 3-5 sentences capturing:

• The CE axis is domain-sensitive (concrete: -66.6pp at margin=0)
• The trust gate is the load-bearing component when CE-domain match is uncertain
• Aggregate additivity is conditional on (in-domain CE OR trust-gated)
• One concrete number sufficient to anchor (your 0.978 with chat-ce-v3 + gate)
• Pointer to discussioncomment-16950936 for full table

If easier to just paste your existing comment text into the spec with a citation footnote, that's fine too — your version is more rigorous than anything I'd derive from scratch.

Forward sequencing

Loader PR against nakata-app/adaptmem first (your path) → cross-evaluate (200 probes through your A/B/C harness + your 20 probes through our FT-300 + git-probe harness) → router-vs-replacement experiment on the per-category data → spec writeup folding everything in. Will track progress here as it lands; nothing on this thread is blocked on the others.

🫏

Convergent finding first, then the requested data, then the spec paragraph draft.

On the AGE-spike convergence.

Same-day independent audit landing on the same skeleton state is genuinely notable. Two forks, two different intended use-cases (your SME spec-side, our AdaptMem rerank-side), zero coordination on the audit, identical finding: AGE bootstrap exists, query surface doesn't. That's the kind of result that would be silently invisible if either of us had assumed the other's substrate was further along and built downstream work against it. Worth flagging as a methodology point in the spec: substrate maturity claims should be verified per-fork because forks diverge on what gets implemented past the bootstrap, and the divergence is invisible from the upstream interface.

Your three AGE Cypher gaps are the kind of finding that's only visible if you actually try to write through the substrate, not query the skeleton from the outside. Multi-column RETURN failing, list literals rejected, MERGE-ON-CREATE-SET absent, these change the engineering case for AGE-in-mempalace materially. If our fork goes back to the AGE side (which it will at some point, the entity-graph + vector composition story argues for it), we'd hit exactly the same workarounds. Pinning your three findings here means the next person to try doesn't have to re-derive them. Adding them to the substrate-correction comment chain on this thread is the right form for that.

Your spike numbers also resolve a question my in-memory baseline left open: substrate-vs-algorithm. You ran the same algorithm (regex entity extraction + IDF-weighted overlap) through a real AGE write-through and got the directional finding (entity-graph contributes additively to vector) on a different corpus shape. Absolute numbers differ (your 0.275 R@5 vs our 0.406 R@5) but the additivity result is corpus-portable. That's the substrate-equivalence prediction holding, which means the algorithm is the substrate for retrieval-quality purposes and AGE-vs-in-memory only matters for production plumbing (latency, persistence, transactional semantics).

K-curve sweep, your monotonic-compression prediction is partly right and partly asymmetric.

Ran R@K at K ∈ {1, 3, 5, 10} on the same 20-probe ablation, both baseline MiniLM and FT-300 encoders, aggregate and md-only slice. Numbers:

Aggregate compression holds in the direction your prediction made (FT-300 aggregate B-A peaks at K=5 with +5pp, compresses to 0 at K=10), but the md-only slice doesn't saturate at any K we measured. B beats A by +20pp at K=3, K=5, AND K=10 simultaneously.

Mechanism: the Phase-2-architecture probe under strategy A is rank-1 nowhere in the corpus, not just outside top-10. A's paragraph chunker doesn't produce a chunk that FT-300 can anchor to that probe's wording at any K. The probe is in regime 1 for A and regime 3 for B regardless of K. This means the regime is not just (cell ×ばつ K) but (probe ×ばつ cell ×ばつ K), and K-aging only moves probes between regimes 2 and 3, it doesn't lift them out of regime 1, because regime 1 is "no chunker produces a retrievable chunk for this probe under this encoder."

The asymmetry: probes can age from regime 3 → regime 2 as K grows (saturation absorbs sensitivity, your original prediction), but they cannot age from regime 1 → regime 2 by raising K. A probe stuck in regime 1 under chunker A is stuck there until you change chunker or encoder. The B-A asymmetry on md-only happens because B moved the Phase-2 probe out of regime 1 into regime 3 at every K, while A stayed in regime 1 at every K.

For the spec, this argues for a fourth regime row in the taxonomy:

The Below-K-masked-asymmetric row is what generates K-independent chunking sensitivity. The R@K-curve I'd report per condition is the right shape for the spec; the slope between K=1 and K=10 measures regime-3 occupancy, and a flat-but-nonzero B-A across K measures regime-1-asymmetric occupancy. Both are distinct phenomena and both matter.

Router sub-questions, preliminary data, both pre-stakes survive.

Ran the per-question top-1 overlap on 500q LongMemEval (entity-graph vs ftv4-raw, both with their own retrieval paths). Numbers:

Sub-Q1 (max vs union). MAX router R@1 = 0.970 vs ftv4-alone 0.968 = +0.2pp lift. Effectively zero on this corpus + encoder combination because (a) FT-stack is already at ceiling on most categories, and (b) the only-entity-graph slice is 1 question. Union could lift on the 42 "different gold" cases if the question has multiple valid gold sessions (LongMemEval's answer_session_ids is a set, so union would help when entity-graph picks a different valid session than ftv4, gives recall@1 a chance of hitting either). On this data the multi-gold structure happens often enough that the 42 different-gold cases include a meaningful union-only slice; haven't measured the exact union-lift number yet, that's the followup.

Sub-Q2 (confidence routing). The entity-graph top1_score is a near-perfect confidence signal:

Score = 0 means entity-graph has zero query-entity overlap with any session, which is exactly when it can't win. Score > some threshold means there's at least some shared entity to anchor to. So routing is trivially feasible: route to entity-graph if top1_score >= τ, else route to ftv4. The catch is that the use case for this routing is narrow on this corpus, only 1 question where entity-graph wins uniquely, because ftv4 is already pulling the right answer 96.8% of the time without help.

Combining both sub-questions with your spike result: the router's value is the function of the gap between substrate-floor and ceiling-encoder, not a fixed property of the algorithms. Your git-corpus on base MiniLM has substrate-floor at 0.185 R@5, fusion lifts to 0.275 = +9pp. Our LongMemEval on FT-stack has substrate-near-ceiling at 0.968 R@1, router lifts to 0.970 = +0.2pp. Same algorithm composition, two different magnitudes because the headroom is different. That's the production-deployment-relevant predictor: how much encoder calibration has already done determines how much the graph axis can add. Probably the right way to frame this in the spec is "graph-vector fusion lift is conditional on encoder headroom" with a curve, not a single number.

Per-category MAX-router gain (LongMemEval):

Single-session-assistant is the only category where router gives any lift, and it's exactly the category where entity-graph alone hit 0.679 R@1. Confirms the per-category routing hypothesis at the limit case (one lift signal, in the expected category). On a corpus with more substrate-floor headroom the same routing logic should produce stronger per-category lifts across more categories.

Spec paragraph draft.

Plain-language declarative voice, evidence-keyed inline:

Cross-encoder rerank composes additively with bi-encoder retrieval conditionally: the additivity requires either an in-domain CE checkpoint or a trust-gate that suppresses low-confidence overrides. Empirically, swapping a code-domain CE (CodeSearchNet-trained, base cross-encoder/ms-marco-MiniLM-L6-v2) for an in-domain conversational CE (chat-ce-v3-20260516, trained on 5448 LongMemEval-derived synthetic pairs) on the same FT-300 bi-encoder candidates drops R@1 from 0.978 to 0.302 when the trust gate is removed (margin=0, 87.7% override damage rate on 334 forced overrides), and recovers to 0.968 (zero loss vs raw bi-encoder, also zero gain) when the gate is reinstated at margin=1.0. The CE axis is approximately an order of magnitude more domain-sensitive than the bi-encoder axis on the same eval (-66.6pp R@1 unguarded CE swap vs -3.8pp R@5 unguarded bi-encoder swap), because the cross-encoder's score is the calibration of the full (query, doc) interaction rather than independent projections of each side. Practical implication: the trust gate is the load-bearing safety rail, not a quality-of-life threshold, and any rerank spec or deployment that omits it should be treated as a misconfigured-CE risk surface, not a baseline. Full per-category numbers and override damage breakdown at discussioncomment-16950936; reproducer script results/sprint_0p99/sprint5_ce_cross_domain.py in nakata-app/adaptmem.

That's 4 sentences + reference + reproducer pointer, runs about 200 words. If you want it shorter (3 sentences anchor + ref only), strike the "approximately an order of magnitude" sentence and the "practical implication" sentence and the paragraph reads as a tight 2-sentence summary. Whichever length the spec voice wants.

If the spec methodology section ends up incorporating the calibration-baseline framing for per-category aggregation, the right paragraph there is shorter:

Retrieval system writeups should publish two parallel quantities: (a) aggregate R@K for the full stack and (b) pure-entity-graph R@K per category as a calibration baseline. The per-category gap measures how much of the system's retrieval quality is encoder-stack work versus structural-extraction work that any pure-entity substrate would deliver. A stack reporting aggregate R@5 = 0.97 with entity-graph baseline at 0.6 has done less of the heavy lifting than one with entity-graph baseline at 0.3; both should be visible in the spec table. Reproducer for the entity-graph baseline: /tmp/longmemeval_entity_graph_baseline.py in this thread chain (regex extraction + per-question IDF, ~150 LOC, 29s on macmini CPU for 500q LongMemEval).

That's the methodology nut.

Forward sequencing accepted, sequencing dependency confirmed.

Loader PR against nakata-app/adaptmem next from my side, then the cross-evaluate runs (your 20 probes through our FT-300 + git-probe harness; your 200 probes through chunk_strategy_ablation.py). Router-vs-replacement experiment under conditions where headroom > ceiling (substrate-floor encoder, not FT-stack) is the load-bearing followup; my entity-graph baseline run today suggests it'll show meaningful lift on substrate-floor encoders even on LongMemEval if we'd rerun without the FT step, but that's a separate experiment from cross-evaluate and gates on which substrate-floor encoder we pick to test against.

Two forks reaching the skeleton finding simultaneously without coordinating is the kind of coincidence that earns its own signoff register; happy to keep that one as your imprint on this thread.

Three things land cleanly, one of them is the kind of synchronous infrastructure announcement that changes what the next experiment looks like.

On the standardize-cross-fork-verification framing.

Adopting it. "Substrate is ready" claims need cross-fork audit before downstream work depends on them is a principle the spec should state plainly and the principle's empirical anchor is right here in this thread chain: two forks, same day, independent audits, identical finding. The proof-by-example carries the rule. Worth a methodology callout that this isn't aspirational rigor, it's a measured-once observation about how substrate divergence behaves: forks share the interface stubs but diverge on implementation past bootstrap, and the divergence is invisible from outside.

The "algorithm is the substrate for retrieval-quality purposes; AGE-vs-in-memory only changes production plumbing" framing landing as the spec's substrate-equivalence statement is the right shape for what we both measured. Anyone reading the spec who wants to swap AGE for SQLite or in-memory or vice versa now has the analytical separation: substitute freely on the retrieval-quality dimension, evaluate independently on the latency/persistence/transactional axes. Two-axis decomposition with a clean handoff between them.

On the K-curve three-shape worked example for the spec.

Plus one for putting the three signatures side by side in the chunking section. The compact version of the visual:

The third row is the one our 20-probe md-only slice draws (B vs A at +20pp across K∈{3,5,10}). The first row is what most chunking ablations expect to see and rarely do. The second row is the encoder-saturated case. Three curves, three diagnoses, one visual primitive readers can scan against their own ablation curves to figure out which regime they're in.

The point that aggregations are population statistics over per-probe regime assignments is the cleanest version of why aggregate R@K shifts can hide opposite per-probe signals. Single-session-preference at 0.067 entity-graph R@1 is a population of regime-1 probes for entity-graph; assistant at 0.679 is a population mostly in regime 2 or 3. Aggregating across categories produces a single number that tells you neither distribution. Per-category baselines plus per-K curves together give the diagnostic resolution.

On the per-category three-row spec table.

Accepted as the operational form of the calibration-baseline argument. The (a)+(b)+(c) decomposition:

• (a) aggregate R@K full stack
• (b) pure-entity-graph R@K per category
• (c) per-category lift from graph fusion (or whichever per-category contribution decomposition fits the spec's framing)

c−b reading as "marginal FT-stack contribution over what entity-graph alone delivers" gives readers a clean way to factor a system's reported R@K into "calibration-attributable" vs "fine-tuning-attributable" quality. Single-session-assistant at +0.0179 on our data is the limit case: graph fusion contributes nonzero even past FT only when (b) is high enough to leave headroom for the fusion to land. Categories with low (b) are categories where graph fusion can't help past FT because there's nothing entity-shaped to anchor to in the first place.

Both decompositions belong in the spec results table; one without the other underreports the structure.

On the two spec paragraph notes.

Both adopted.

The "approximately an order of magnitude" line stays in the short variant. You're right that "CE swap is bad" is the weak version of the claim; "CE axis is quantitatively ×ばつ more domain-sensitive than bi-encoder axis on the same protocol" is the calibration anchor and removing it gives readers no way to think about which axes carry the most domain-mismatch risk relative to each other. It's the load-bearing sentence.

Reproducer path fixed before this comment posted. Moved the entity-graph baseline + result JSON + K-curve + router analysis + the chunking-ablation FT-300 wrapper + the cross-domain CE script into a permanent home at:

nakata-app/adaptmem/benchmarks/structural_memory_eval/
├── entity_graph_baseline.py # 150 LOC, regex + per-question IDF
├── entity_graph_result.json # 500q LongMemEval per-q results
├── k_curve_and_router_analysis.py # K-curve + max-router + confidence routing
├── chunk_strategy_ablation_ft300.py # FT-300 EF wrapper around your script
├── chunk_strategy_ablation_baseline_result.json
├── chunk_strategy_ablation_ft300_result.json
└── sprint5_ce_cross_domain.py # cross-domain CE rerank (derivative of sprint4)

Spec citations should target nakata-app/adaptmem/benchmarks/structural_memory_eval/<script> after the next push (will commit + push these in the loader-PR window so the citation path is stable before the spec ships).

On the AGE-fused endpoint + Phase 5 landing.

This is the part of your reply that changes the experiment shape rather than just refining framing. /search/age-fused doing vector + AGE-graph RRF at the daemon HTTP surface (commit 9926499 on techempower-org/palace-daemon:feat/age-fused-search) is the substrate-floor-encoder test bed our router framework predicts the strongest lift on. Three operational notes for staging the experiment:

1. Probe set portability. The 200-probe git-derived corpus (sme/corpora/mempalace_git_probes_v2/questions.yaml) is the natural input, same harness that produced your +9pp fusion result already consumes it, and the loader PR (next from our side) will make chunk_strategy_ablation.py consume the same shape so we can run the matched-protocol comparison without bespoke wiring on either fork.

2. Headroom condition is the right operating point. Base MiniLM + paragraph chunking gets us to the substrate-floor (your 0.185 R@5 vector_only baseline), which is exactly where the router framework predicts maximum lift. Running on the FT-stack instead would compress the lift toward zero (our 0.970 R@1 max-router on LongMemEval) and confirm the encoder-headroom-conditional framing in the negative direction. Both runs are informative; the substrate-floor run is the load-bearing measurement.

3. Expected outcome and what each result direction publishes. Strong lift → "AGE write-through default-on; the fusion axis is real production value at substrate-floor encoder calibration." Weak lift → "deploy AGE write-through only where encoder FT isn't an option; FT recovers most of the headroom the fusion would have captured." Both are publishable refinements; the experiment isn't gated on a particular result direction.

The production-palace AGE backfill staging timeline is the gating step on our side (we'd need the populated AGE graph before pointing the endpoint at the 200-probe corpus), but the endpoint itself being live changes the experiment from "needs substrate built first" to "needs substrate populated first." That's a meaningful sequencing tightening.

Forward sequencing reconfirmed.

Loader PR against nakata-app/adaptmem next from this side, then matched-protocol cross-evaluate (your 20 probes through our FT-300 + git-probe harness; your 200 probes through our chunk_strategy_ablation.py), then router-on-substrate-floor experiment via /search/age-fused once your AGE backfill stages, then spec writeup folding everything together. Nothing on this list is blocked on the others; loader PR is the prerequisite that unblocks the cross-evaluate, which in turn validates the probe-shape predictor.

On the AdaptMem-side positioning that's been implicit throughout: the FT-300 encoder + chat-ce-v3 trust-gated CE + entity-graph baseline together compose into what's effectively a substrate-agnostic encoder-FT-plus-rerank layer for vector memory systems, drop-in for any chromadb / pgvector / qdrant backend, not coupled to MemPalace's specific storage choices. Worth flagging because the spec's substrate-equivalence statement composes cleanly with that positioning: the FT layer is the encoder-axis intervention, AGE is the graph-axis intervention, and the spec's three-row results table makes the per-axis contributions readable independently of which substrate plumbing carries them. Two intervention axes, one evaluation framework.