Skip to content

Navigation Menu

Sign in
Sign up

perf(whisper): reuse the language-detect encoder pass (1.9x on auto) - #31

Open
sumerc wants to merge 15 commits into
main from
improve-whisper-auto-latency
Open

perf(whisper): reuse the language-detect encoder pass (1.9x on auto) #31
sumerc wants to merge 15 commits into
main from
improve-whisper-auto-latency

Conversation

@sumerc

@sumerc sumerc commented Aug 6, 2026

Copy link
Copy Markdown
Owner

What

In auto-detect mode whisper_full encoded the same audio twice: once inside
whisper_lang_auto_detect_with_state, then again for the first decode window.
On dictation-length clips the encoder is the cost, so auto-detect simply
doubled it.

The patch tags the encoder output with the (mel_offset, n_audio_ctx) it was
computed from and skips a re-encode that would reproduce it, and assigns
exp_n_audio_ctx before detection so one call encodes at one window size.

Numbers

M5 Pro, internal/localbench, best of 5, whisper-turbo-q5:

clip auto before auto after speedup forced lang
short (1.4 s) 530 ms 274 ms 1.94x unchanged
en (1.6 s) 538 ms 278 ms 1.94x unchanged
en-5.2s 538 ms 278 ms 1.94x unchanged
tr-9.8s 588 ms 329 ms 1.79x unchanged
en-70s 1383 ms 1137 ms 1.22x unchanged
en-183s 3343 ms 3075 ms 1.09x unchanged

Transcripts are unchanged — the skipped work was bit-identical, and auto output
now equals forced-language output on every corpus clip. Long clips gain less
because one saved encode amortises over many windows.

Why it is guarded twice

Every failure mode here is silent: an unpatched build is still correct, just
2x slower on auto, and no test can tell a correct-but-slow build from a fast
one. git apply also only matches context lines, so after an upstream bump the
patch could apply cleanly onto a restructured encode path and quietly stop
working. So:

  • make whisper-lib refuses to build unless the submodule HEAD is the
    WHISPER_BASE commit the patch was benchmarked against.
  • TestWhisperPatchesApplied compares the checkout's diff to the patch
    byte-for-byte, catching a dropped patch, a bump that shifted hunks, and
    hand-edits that ignore = dirty hides.

Upstream

This is ggml-org/whisper.cpp#3954, opened from this project and still unmerged,
now implemented — and more generally than the issue proposed: keying on
(mel_offset, n_audio_ctx) holds for every window, not just the first, which
drops the issue's caveats about nonzero offsets and custom audio_ctx. When it
merges upstream, delete the patch file and bump the pin.

A fork carrying the patch as a commit was considered and rejected while it is a
single upstream-bound patch — rationale in docs/design-notes.md.

Side effects

Fixes fault-matrix case H (cold state, auto, sized window), which reopens
audio_ctx sizing as a quality decision rather than a correctness one.
Sizing is worth a further ~1.7x but is not transcript-preserving, so it is
not taken here: audioCtxFor still returns 0.

Test plan

  • go test ./internal/whisper — including TestTranscribeKnownClips
  • ZEE_AC_DEBUG=1 ... -run FaultMatrix — case H now passes, D/F/G/I/J/L unchanged
  • internal/localbench before/after on the full corpus (table above)
  • patch reapplies after git submodule update resetting the checkout
  • non-applying patch and wrong WHISPER_BASE both fail the build hard
  • TestWhisperPatchesApplied fails on a hand-edit to the submodule
  • not yet measured on M1/M2 — the win is structural (one encoder pass instead
    of two) so it should transfer, but that is inference, not measurement

🤖 Generated with Claude Code

sumerc and others added 2 commits August 6, 2026 20:12 
whisper_full encoded the same audio twice in auto mode: once inside
whisper_lang_auto_detect_with_state, then again for the first decode
window. Tag the encoder output with the (mel_offset, n_audio_ctx) it was
computed from and skip the identical re-encode; assign exp_n_audio_ctx
before detection so one call encodes at one window size.
Auto-detect now costs the same as a forced language: 1.94x on
dictation-length clips (530 -> 274 ms, M5 Pro, turbo-q5), with
transcripts unchanged — the skipped work was bit-identical. The
forced-language path never had the second encode and is untouched.
Upstream as ggml-org/whisper.cpp#3954, unmerged, so it lives in
patches/whisper.cpp and make whisper-lib applies it.
Guarded twice, because an unpatched build is still correct and merely 2x
slower on auto, which no test can distinguish from a fast one:
WHISPER_BASE stops the build on a submodule bump, and
TestWhisperPatchesApplied matches the checkout against the patch
byte-for-byte (also catching hand-edits that ignore = dirty hides).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What was measured while landing the encoder-reuse patch, so none of it
gets re-derived: the before/after numbers, whisper's own time budget
(the encoder is 94% of a dictation transcribe and flat in clip length),
and the ggml 0.13 -> 0.18 bump measuring neutral on M5 Pro.
Also corrects two entries the patch invalidates. The audio_ctx rejection
is partly superseded: fault-matrix case H no longer garbles and
whisper_init_state is ~10 ms, not the Metal setup it was assumed to be,
so sizing is now a quality call rather than a correctness one. The
silence-trimming entry gains a conditional, since "not a speed lever"
holds only while the encoder window is fixed.
Adds the alternative-engine survey: the search space collapses because
the 30 s padding is a Whisper architecture property, not a whisper.cpp
one. Notes that Parakeet's coverage excludes Turkish and every
non-European language, so for those users Whisper is the only local
engine and audio_ctx sizing is the only remaining lever; that upstream
whisper.cpp now ships Parakeet but with an incompatible model format, so
migrating costs a model release and buys build simplicity only; and that
MLX is the one untested alternative that survives the coverage filter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sumerc sumerc self-assigned this Aug 6, 2026
sumerc marked this pull request as ready for review August 6, 2026 22:48

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new blocking findings · 2 minor points

🔍 Full review · 6 files reviewed

🔵 Minor points

Not blocking, and no threads opened for these.

  • internal/whisper/patch_test.go:45 — TestWhisperPatchesApplied builds its expected diff by concatenating each *.patch file in os.ReadDir (filename-sorted) order and compares it byte-for-byte against a single git diff of the whole submodule. git diff emits one unified diff ordered by source path and merges all hunks touching the same file under a single diff --git header. This matches today's single patch, but a second patch — which the design notes explicitly anticipate ("if the patch set grows past two or three") — makes the two orderings diverge: two .patch files whose filename order differs from source-path order, or two patches touching the same source file, will not equal git's single combined diff. The test then fails even when the checkout is correctly patched.
  • internal/whisper/whisper.go:153 — The new audioCtxFor comment embeds machine-specific performance numbers ("measured ~10 ms", "worth a further ~1.7x") without naming the machine, whereas the repository convention keeps such numbers out of code comments and, when one must appear, requires naming the machine (e.g. "~10 ms on an M5 Pro"). The measurement is already recorded in docs/design-notes.md with the M5 Pro label, so the comment can cite it without the bare figure.
Review details
  • Commit: c1bdb7e
  • Model: claude-opus-4-8
  • Panel: correctness · robustness · design

// mercy of the developer's diff config.
cmd := exec.Command("git", "-c", "core.pager=cat", "-C",
filepath.Join(root, "third_party", "whisper.cpp"),
"diff", "--no-color", "--no-ext-diff")

@upsun-dispatch-dev upsun-dispatch-dev Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning — Spurious build/test failure on a correct tree under a non-default global git config.

The comment above this call claims the fixed flags keep the comparison from being "at the mercy of the developer's diff config," but the command pins only --no-color/--no-ext-diff (and core.pager=cat, which cmd.Output() makes irrelevant). git diff still emits the index <old>..<new> line abbreviated per core.abbrev and the a//b/ path prefixes per diff.noprefix/diff.mnemonicPrefix. The stored patch uses 8-char hashes and a/,b/ prefixes; a developer or CI runner with a global core.abbrev = 12 (or diff.noprefix = true) produces a differently-formatted diff, so string(out) != want.String() fires and the test t.Fatalf's on a correctly-patched checkout. Pin these explicitly (e.g. -c core.abbrev=8 -c diff.noprefix=false -c diff.mnemonicPrefix=false). A related latent issue: because git diff merges all hunks for one file under a single diff --git header while the concatenated patch files each carry their own header, adding a second patch that also touches src/whisper.cpp (the PR anticipates growing to two or three) would make this comparison mismatch even when both patches apply correctly.

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new blocking findings · 2 minor points · 1 still open

🔁 Incremental · 8 files reviewed

🔵 Minor points

Not blocking, and no threads opened for these.

  • internal/whisper/whisper.go:289 — The Transcribe doc comment just above transcribeAt still states that "" auto-detect "is the only mode that survives code-switching — a wrong forced language garbles the output rather than merely mislabelling it." This PR's own new comment in transcriber/whisper.go and the new design-notes section ("Why English is the default language for every model") retract both halves: auto is the mode that breaks on mixed audio, and a wrong forced language translates fluently rather than garbling. The two comments in the same change now contradict each other, so the next reader of internal/whisper/whisper.go gets the opposite of the rationale the rest of the PR documents.
  • internal/whisper/whisper.go:53 — zee_wsp_det_lang / zee_wsp_det_prob are process-global statics written by the single process-wide whisper_log_set callback and read by LastDetection(). The comment claims "only one call is ever in flight" under "the Ctx mutex", but that guarantee is per-Ctx: two different Ctx instances transcribing concurrently (e.g. a warm-up New() overlapping another transcribe, or tests creating multiple contexts) write/read these globals with no shared lock, racing on the diagnostic result. Production is single-model and serial so it does not hit this today, but the invariant the comment asserts is process-wide, not per-Ctx.
Review details
  • Commits: 8e11dae..61f8ca4
  • Model: claude-opus-4-8
  • Panel: security · correctness · robustness · design

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

upsun-dispatch-dev Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

📋 PR Summary

This PR speeds up local Whisper auto-detect transcription by reusing the encoder output already computed during language detection instead of encoding the audio a second time, delivering roughly a 1.9x win on short clips. It carries the change as an in-tree patch against a pinned whisper.cpp submodule, guarded by a WHISPER_BASE build pin and a byte-for-byte patch-verification test. Alongside it, the PR adds a deterministic post-transcription vocabulary corrector (correct/) that maps misheard spans onto the hints.txt dictionary, extends hints with spoken-form alias syntax, and records extensive supporting measurements in the design notes.

Changes
Layer / File(s) Summary
Whisper encoder-reuse optimization
patches/whisper.cpp/0001-reuse-detect-encoder-output.patch In-tree patch tagging the encoder output with its (mel_offset, n_audio_ctx) so a redundant re-encode after language detection is skipped.
internal/whisper/whisper.go Whisper cgo bindings updated to support the encoder-reuse path and language-detection reporting.
internal/whisper/whisper_stub.go Non-darwin/arm64 stub extended with matching signatures (LastDetection) so the no-cgo build stays in sync.
internal/whisper/patch_test.go New test comparing the submodule diff byte-for-byte against patches/whisper.cpp/*.patch to catch dropped, drifted, or hand-edited patches.
Makefile whisper-lib target pins WHISPER_BASE, applies missing patches, and refuses to build against an unexpected submodule HEAD.
.gitmodules Adds the whisper.cpp submodule with ignore = dirty (the patch leaves the checkout permanently dirty).
Vocabulary correction
correct/correct.go New package applying deterministic fuzzy/exact vocabulary correction (Levenshtein plus Soundex boost, acronym spoken-form expansion) with common-word and non-ASCII guards.
correct/correct_test.go Tests covering corpus mishearings, n-gram merges, common-word protection, alias parsing, and case/punctuation preservation.
config/hints.go Adds HintLines() returning raw alias-bearing hint lines for the corrector, and documents the Term: alias syntax; alias part is stripped before sending hints to providers.
main.go Wires the corrector into the transcription flow with a -no-correct flag and logs applied corrections.
main_test.go Tests for the new correction wiring and flag handling.
Transcriber plumbing
transcriber/local.go Local engine updates supporting language detection and the encoder-reuse path.
transcriber/local_session.go Logs the detected language after inference via the engine's LastDetection hook.
transcriber/whisper.go Whisper transcriber adjustments for detection reporting.
transcriber/deepgram.go Provider adjustments related to hints/correction handling.
transcriber/groq.go Provider adjustments related to hints/correction handling.
transcriber/mistral.go Provider adjustments related to hints/correction handling.
transcriber/openai.go Provider adjustments related to hints/correction handling.
transcriber/warm_test.go New test around connection warming.
Tray cleanup
tray/tray.go Removes now-unused tray code.
tray/tray_darwin.go Removes now-unused darwin tray code.
tray/tray_other.go Removes a now-unused line in the non-darwin tray stub.
Docs & config
docs/design-notes.md Extensive new entries on the encoder-reuse decision, corrector safety evaluations, multilingual damage, Soundex vs IPA, parakeet auto-detect, and turbo mis-detection; also revises the facts-vs-decisions guidance.
CLAUDE.md Documents -no-hints/-no-correct flags and the updated design-notes maintenance rule.
.gitignore Ignores internal experiment directories (hypo-eval/, review notes).

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new blocking findings · 2 minor points · 1 still open

🔁 Incremental · 5 files reviewed

🔵 Minor points

Not blocking, and no threads opened for these.

  • transcriber/local.go:226 — Warm() ignores the error from eng.Transcribe (line 222) and then unconditionally sets p.lastUsed = time.Now(). When the warm pass fails — a transient engine error, or the engine being Closed between the ready-check and the call — the idle timer is reset even though no weight pages were faulted in, so no further warm is attempted for another warmIdleThreshold (10 min) and the next real dictation pays the full cold page-in with no diagnostic logged.
  • transcriber/local.go:221 — The comment justifies forcing lang "en" by claiming an auto warm pass "would log a bogus lang_detect line for silence." Warm calls eng.Transcribe directly rather than through localSession, and logDetectedLanguage — the only emitter of that log line — runs in localSession.Close(), which the warm path never reaches. No lang_detect line is logged regardless of the language passed; the real reason to force a language is skipping the wasted detection encode.
Review details
  • Commits: 61f8ca4..3144a1c
  • Model: claude-opus-4-8
  • Panel: correctness · robustness · design

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new issues · 1 still open

🔁 Incremental · 1 file reviewed

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

Review 5 of 10 for this pull request · View the full run

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Changes suggested — 🟡 1 warning · 4 minor points · 1 still open

🔍 Full review · 19 files reviewed

🔵 Minor points

Not blocking, and no threads opened for these.

  • main.go:1518 — The applyCorrection function and its doc comment were inserted directly beneath the existing // runTranscribeFiles transcribes one or more files ... comment (lines 1515-1517) but above func runTranscribeFiles (line 1533). As a result that comment now stacks on top of applyCorrection, and runTranscribeFiles is left with no doc comment at all.
  • config/hints.go:104 — HintLines() opens hints.txt read-only and returns nil if it does not exist, unlike GetHints() which auto-creates it from the default template. When the app is only ever run with -no-hints, config.SetHints("") pins hints so GetHints() returns early and never creates the file; on a config dir that was never otherwise initialized, hints.txt therefore never exists, HintLines() returns nil, and applyCorrection parses an empty dictionary. That silently disables the corrector — contradicting this function's own doc comment that -no-hints "must not disable post-transcription correction."
  • correct/correct.go:147 — Once the dictionary is non-empty, Apply runs strings.Fields(text) then strings.Join(result, " ") unconditionally, so it reflows whitespace on every English transcription even when no term is corrected: runs of multiple spaces collapse to one and newlines/leading/trailing whitespace are dropped. finishTranscription now calls this on every English transcript with a non-empty hints.txt, so text like "Section 1.\n\nOverview" comes back as "Section 1. Overview" with no correction applied. Word-space normalization is documented on Apply, but newline/blank-line stripping is not.
  • main.go:22 — The stdlib import "strings" was added inside the zee/* import group (between "zee/clipboard" and the blank line before "zee/config") and is out of alphabetical order within that group, so gofmt -l/goimports reports main.go as unformatted.
Review details
  • Commit: fe16254
  • Model: claude-opus-4-8
  • Panel: security · correctness · robustness · design

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

Review 6 of 10 for this pull request · View the full run

Comment thread main.go Outdated

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new issues · 1 still open

🔁 Incremental · 4 files reviewed

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

Review 7 of 10 for this pull request · View the full run

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Reviewed — No new blocking findings · 1 minor point · 1 still open

🔁 Incremental · 9 files reviewed

🔵 Minor points

Not blocking, and no threads opened for these.

  • docs/design-notes.md:1553 — This increment deletes SupportsHints (transcriber/local.go) and switches whisper/Groq to never send hints, but the earlier section "## Why hints reach Whisper but not Parakeet" (line 1171) still states the opposite: "Whisper takes the same comma-separated string ... as initial_prompt" and "the gate is now SupportsHints, a per-engine capability." Both are now false — the symbol no longer exists anywhere in the tree and whisper providers no longer send the prompt field. That section is not marked superseded, unlike the other reversed entries in this file (e.g. lines 47, 74, 205). The repo convention requires marking a superseded entry rather than leaving it to contradict the new note added here.
Review details
  • Commits: 8b31816..6c72f1a
  • Model: claude-opus-4-8
  • Panel: correctness · design

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

Review 8 of 10 for this pull request · View the full run

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Changes suggested — 🟡 1 warning · 1 still open

🔁 Incremental · 3 files reviewed

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.

Review 9 of 10 for this pull request · View the full run

Comment thread main.go
for i, r := range reps {
pairs[i] = r.From + "→" + r.To
}
log.Info(fmt.Sprintf("corrections n=%d [%s]", len(reps), strings.Join(pairs, ", ")))

@upsun-dispatch-dev upsun-dispatch-dev Bot Aug 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning — Leaks fragments of user speech into an always-on log, bypassing the -debug-transcribe opt-in.

This log.Info call writes each correction as From→To into diagnostics_log.txt, which is always on. r.From is a verbatim span of the user's transcribed speech (the misheard words that fired a correction). The project keeps transcript text out of the always-on log and confines it to transcribe_log.txt, created only when -debug-transcribe is passed (log.TranscriptionText guards on transcribeFile != nil). Logging the spans here bypasses that gate: any clip that triggers a correction leaves fragments of what the user said in a log they never opted into for transcript capture. Log only the dictionary terms (r.To) here, or route From through the -debug-transcribe-gated channel.

@upsun-dispatch-dev upsun-dispatch-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Changes suggested — 🟡 1 warning · 1 minor point · 2 still open

🔁 Incremental (head + base moved) · 3 files reviewed

🟡 Warning (P1 — real issues) — outside diff range

docs/design-notes.md · merge-conflict

This PR does not merge cleanly onto the current base branch: docs/design-notes.md conflicts. The base moved (commit 2d4f0c4 appended an "install.sh resolves the release tag from a redirect" entry at the end of the file) while this PR also appends new entries to the same file, so the two edits collide. The PR must be rebased and the docs/design-notes.md conflict resolved before it can merge.

🔵 Minor points

Not blocking, and no threads opened for these.

  • docs/design-notes.md:1635 — The new corrector-eval entries repeatedly describe "the shipped 28-term hints.txt" (also lines 1710 and 1737), but the committed default in config/hints.go lists 18 terms (Opus, Claude, Sonnet, Fable, Pi.dev, JSON, Codex, Harness, Gzip, OpenAI, Anthropic, App Router, Grafana, favicon, Mistral, ElevenLabs, Bun, Node.js). A reader who cross-checks the shipped file against the note finds the count does not match — the eval table (28) and the shipped file (18) are being conflated under the word "shipped".
Review details

Outstanding from earlier reviews:

  • #3735008296 — internal/whisper/patch_test.go:55: Spurious build/test failure on a correct tree under a non-default global git config.
  • #3790376160 — main.go:1539: Leaks fragments of user speech into an always-on log, bypassing the -debug-transcribe opt-in.

Review 10 of 10 for this pull request · View the full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@upsun-dispatch-dev upsun-dispatch-dev[bot] upsun-dispatch-dev[bot] left review comments

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /