Watch 1

Star 11

HTTPS

Download ZIP Download TAR.GZ Download BUNDLE Open with VS Code Open with VSCodium Open with Intellij IDEA

Read this in other languages: 韓國語 (Korean).

---

Bibim

Bibim is an input method for CJK (Chinese, Japanese, Korean) languages that treats Chinese characters as a shared bridge between writing systems. Where conventional input methods assume a single language at a time, Bibim lets users draw on vocabulary from multiple CJK languages simultaneously, entering words through whichever phonetic reading they happen to remember and receiving output in whichever script and orthographic convention they need.

The primary goal is mixed-script Korean (國漢文混用體), where Sino-Korean vocabulary is written in Chinese characters alongside native Korean in hangul. Japanese and Chinese (both simplified and traditional) are secondary targets, but the architecture is designed to handle all three language families through a single, unified pipeline.

Motivation

Speakers of CJK languages share an enormous pool of Sino-vocabulary. The word for "museum" is 博物館 across Korean, Japanese, and Chinese; only the pronunciation differs (Korean bangmulgwan, Japanese hakubutsukan, Mandarin bówùguǎn). Conventional input methods work within a single language at a time: a Japanese input method accepts Japanese readings, a Korean one accepts Korean readings. Bibim takes a different approach. If you are writing Japanese and cannot recall the Japanese reading of 博物館, you can type the Korean reading instead, or the Mandarin reading. The system resolves the characters through the shared logographic layer and outputs them in whatever language and script variant you have selected.

This is not translation. The word 博物館 exists independently in all three languages. Bibim simply lets you reach it from any direction.

A secondary motivation is mixed-script Korean input. Modern Korean is written almost entirely in hangul, but a tradition of mixed hanja-hangul writing persists in academic, legal, and literary contexts. Conventional Korean input methods handle hanja conversion at the single-character level. Bibim approaches this differently, treating mixed-script output as a first-class mode: it converts entire Sino-Korean words to hanja at once, with candidate ranking informed by frequency data.

The three layers

Bibim's interface is organized around three independent axes of configuration. They are best understood as layers rather than modes, since each one governs a different stage of the pipeline.

Input script

The writing system the user physically types in: hangul, Latin alphabet, kana, bopomofo, and so on. This layer does not concern itself with physical key mappings or hangul jamo composition; those are handled separately by the operating system or a lower-level keyboard driver. The choices here are limited to phonetic or syllabic scripts.

Vocabulary language

The language (or languages) whose dictionary is consulted when generating candidates. This can be set to more than one language at a time. If set to both Korean and Japanese, a hangul input of "박물관" matches the Korean reading of 博物館, while "하쿠부츠칸" matches the Japanese reading of the same characters. Both resolve to the same logographic form but through different phonetic paths.

Output language

The target rendering policy of the final output. This is not just a language choice; it also fixes the specific character variant set (字體) to use. A variant set is a systematic convention for writing Chinese characters within a particular language or standard: shinjitai, simplified, traditional, and so on. In implementation terms, the pipeline resolves this choice into one global target output variant that applies to the whole candidate, not a per-segment setting tied to each matched lexeme's source vocabulary language. Options include:

• Korean, mixed script (hanja where applicable, hangul elsewhere)
• Korean, hangul only
• Japanese, shinjitai (新字体)
• Japanese, kyūjitai (舊字體)
• Chinese, simplified (简化字)
• Chinese, traditional (繁體字)

The output language determines the final rendering of every segment in the candidate. If the vocabulary lookup resolves to 博物館 and the output language is set to Korean (hangul only), the output is 박물관. If set to Japanese (shinjitai), the output is 博物館 as-is (no simplification needed in this case). If set to Chinese (simplified), 圆周率 rather than 圓周率. The same principle applies when the matched lexeme comes from another vocabulary language: a Korean lexeme normalized as 大學 still renders as 大学 when the selected output language is Japanese (shinjitai).

How it works

A concrete example. The input script is hangul, the vocabulary languages are Korean and Japanese, and the output language is Japanese (shinjitai). The user wants to write 博物館に行きましょう ("let's go to the museum").

They cannot remember the Japanese reading of 博物館. So they type the Korean reading instead:

박물관니

They press Space to convert. Bibim's pipeline does the following:

1. The input string "박물관니" is segmented against the dictionary and the transliteration rules. One strong dictionary match from the Korean vocabulary is 博物館 (reading: 박물관). The remaining span "니" is transliterated to the kana に.
2. The candidate 博物館に is presented, with the matched logographic segment normalized to the canonical form.
3. The canonical form is converted to shinjitai for output. In this case no change is needed, so the output is 博物館に.

The user selects the candidate. They then type the rest of the sentence phonetically:

이키마쇼우

This is a hangul approximation of いきましょう. The transliteration layer produces a set of possible kana strings (because the mapping from hangul to kana is not always one-to-one; for instance, "가" could map to either か or が). Each candidate is matched against the dictionary. In the current bundled dataset, the winning path is the stem-plus-auxiliary segmentation 行き + ましょう; nonsense combinations do not survive, and the rendered output is still 行きましょう.

The final output: 博物館に行きましょう.

Interactive conversion flow

In practice, conversion is not a single batch operation. The pipeline runs incrementally as the user types, updating the preedit and candidate list after each keystroke.

Continuing the same scenario. The user begins typing in hangul:

• "박": No dictionary match yet. The preedit shows 박 (a passthrough fallback). The reading index finds 博物館 as a prefix match for "박물관" and offers it as a predicted candidate.
• "박물": The prediction 博物館 rises in the candidate list. An exact match for 博物, if one exists in the dictionary, also appears.
• "박물관": An exact dictionary match for 博物館 appears and outranks the prediction, becoming the top candidate.
• "박물관니": The lattice now contains the path 博物館 + に. This becomes the top candidate.

The user selects 博物館に and commits it. The pipeline saves the committed context: the last edge's vocabulary language, connection ID, and transition model. This context is an immutable value, not hidden mutable state.

The user then types "이키마쇼우". A new lattice is constructed from scratch for this input, but the committed context feeds into the transition cost of the first edge, so the new input is scored as a continuation of 博物館に. The winning path 行き + ましょう is matched and committed. The final output is the same: 博物館に行きましょう.

In the current prototype, this interaction boundary is factored into the packages/session/ package. It defines the typed session snapshot, reducer, and shell-facing effects. The shell-neutral assembly of presets, runtime configuration, and SessionEnvironment now lives one layer up in packages/prototype-runtime/, which the web demo consumes as a thin shell.

This shell-to-engine boundary is being refined into a typed, serializable async protocol. Candidate generation, session reduction, and optional max-package loading will run behind an asynchronous boundary—a browser worker in the prototype, a Rust host in production—while the shell retains only platform-specific concerns such as DOM focus, IME composition, input scheduling, and URL state. The engine becomes the authoritative owner of session state and answers requests with declarative snapshots rather than imperative effect streams, so the same contract survives transport changes. The detailed protocol design is in packages/prototype-runtime/docs/async-engine-boundary.md.

Pipeline architecture

The pipeline has three stages. Each operates as a pure function over immutable data.

flowchart LR
 A["Raw input<br>(hangul, kana, etc.)"]
 B["Lattice construction + Viterbi<br>(transliteration ×ばつ dictionary)"]
 C["Output conversion<br>(candidate path → target output variant)"]
 A --> B --> C

Lattice construction with per-span transliteration

The core of candidate generation. The system scans the input string and, for each possible span, invokes transliteration to produce candidate readings, then matches those readings against the dictionaries of all selected vocabulary languages. Each successful match becomes an edge in a lattice (a directed acyclic graph) whose nodes are positions in the input string.

Transliteration is not a preprocessing step that converts the entire input into a single phonetic representation before dictionary lookup. It is an edge generator, invoked per span and per vocabulary language during lattice construction. This matters because a single input string can contain spans that belong to different languages. In the input "박물관니", the span "박물관" is consumed as a Korean hanja reading (producing the edge 博物館), while "니" is consumed as a Japanese kana transliteration (producing the edge に). These two edges come from different vocabulary languages and different transliteration rules, but they coexist in the same lattice.

transliterate(
 span: string,
 inputScript: Script,
 vocabLanguage: Language,
) -> Set<string>

The return type is Set<string> because the mapping between scripts is sometimes ambiguous. Korean hangul "가" might correspond to Japanese か or が; both are returned, and the ambiguity is resolved by dictionary matching. The transliteration layer can be permissive without sacrificing precision: it generates all plausible readings, and the dictionary filters out the ones that do not correspond to real words.

Dictionary matching is performed against a reading index of surface forms, not only citation headwords. The reading index is structured as a trie, so that both exact match and prefix lookup are resolved in a single traversal. Exact matches produce dictionary-match edges; prefix matches produce prediction edges (see below). An inflecting lexeme such as Japanese 行く can therefore contribute multiple surface forms or segment-compatible pieces to the lattice. The current bundled dataset, for example, resolves ikimashou as 行き + ましょう rather than as one monolithic 行きましょう lexeme, while still pointing both pieces back to stable lexeme metadata. The prototype can materialize whole-word surface forms during dictionary construction where that is useful; a later production system may replace part of this with rule-based generation without changing the lattice interface.

Not all input script–vocabulary language combinations are equally productive. Identity and well-established mappings (hangul→Korean, kana→Japanese, Latin→Japanese via romaji, Latin→Chinese via pinyin) produce tight candidate sets. Combinations that cross wider phonological gaps (hangul→Japanese, hangul→Mandarin) are lossier: transliteration returns larger sets, and the dictionary must filter more noise. The architecture tolerates this—the cost is wasted dictionary lookups, not incorrect output—but practical utility diminishes with phonological distance.

In addition to dictionary-matched edges, the lattice includes single-character passthrough edges for every position. A passthrough edge carries the raw input character forward (a hangul syllable, a kana, a Latin letter) without any dictionary match. Its cost is set very high, so the Viterbi algorithm prefers dictionary-matched paths whenever they exist. Passthrough edges guarantee that the lattice always contains at least one complete path from start to end, even when parts of the input match nothing in any dictionary.

In addition to exact-match and passthrough edges, the lattice supports prediction edges. When the reading index trie finds that a typed span is a proper prefix of a longer reading, it produces a prediction edge. This edge consumes only the input characters actually typed, but represents the full dictionary word that the prefix points to. Prediction edges are only allowed as the last edge in a path, because they fill in characters the user has not yet typed; placing them mid-path would produce incoherent output by mixing predicted characters with subsequent real input. A prediction edge carries a cost penalty proportional to the number of untyped syllables remaining. As the user continues typing and more of the reading is supplied, exact-match edges appear and outrank predictions through the normal cost competition, without any special-purpose logic.

Cross-language lattice example

Consider the input "박물관니" with vocabulary languages set to Korean and Japanese:

graph LR
 P0((0)) -->|"博物館 [ko]"| P3((3))
 P3((3)) -->|"に [ja]"| P4((4))
 P0 -->|"박 ?"| P1((1))
 P1 -->|"물 ?"| P2((2))
 P2 -->|"관 ?"| P3

The edge 0→3 comes from the Korean vocabulary (reading "박물관" matches 博物館). The edge 3→4 comes from Japanese kana transliteration ("니" maps to に). Single-syllable fallback edges (marked with ?) exist but carry high cost. The Viterbi search finds the minimum-cost path: 博物館 [ko] + に [ja].

This is what distinguishes Bibim from a conventional single-language input method: edges from different vocabulary languages coexist in the same lattice, and the search finds the best path across all of them.

Single-language lattice example

For a purely single-language case, the lattice works the same way as in a conventional input method. Given the input "대학생활" (Korean reading) and a dictionary containing 大, 大學, 大學生, 學, 學生, 生, 生活, and 活:

graph LR
 P0((0)) -->|大| P1((1))
 P0 -->|大學| P2((2))
 P0 -->|大學生| P3((3))
 P1 -->|學| P2
 P1 -->|學生| P3
 P2 -->|生| P3
 P2 -->|生活| P4((4))
 P3 -->|活| P4

A greedy longest-match strategy would pick 大學生 + 活, but the Viterbi algorithm, informed by word frequency, correctly prefers 大學 + 生活 ("college life").

Incremental lattice extension

The lattice supports incremental construction as the user types. When a character is appended, the existing edges remain valid; only edges ending at the new position need to be added. These are the edges whose spans start at any earlier position and end at the newly typed character, requiring O(n) new span evaluations where n is the current input length.

The Viterbi forward pass extends incrementally as well. The optimal path values for all positions before the new character are unchanged, so only the new position needs to be evaluated against its incoming edges. Per-position Viterbi state is retained as a stack-like sequence of immutable snapshots.

Deletion (Backspace) pops the last position's edges and Viterbi state in O(1). Arbitrary edits (insertion in the middle, bulk replacement) fall back to full recomputation of the lattice and Viterbi state from scratch.

This incremental model does not compromise the pipeline's pure-function character. Each step produces a new immutable lattice and Viterbi snapshot from the previous one and the newly typed character; no hidden mutable state accumulates across keystrokes.

Cost model

The basic cost model assigns each edge a cost derived from word frequency: common words have low cost, rare words and single-character fallbacks have high cost. With unigram edge costs, the Viterbi algorithm computes the minimum-cost path through the lattice in O(E) time, where E is the number of edges. Lattice construction itself enumerates O(n^2) spans for an input of n characters, invoking transliteration and dictionary lookup for each span and vocabulary language; since inputs are short (a word or phrase before the user presses Space), this is fast in practice.

Bibim then adds a small amount of finite-state context on top of this base model. First, a cross-language transition cost: when multiple vocabulary languages are active, switching languages between adjacent edges incurs a penalty. This does not shrink the lattice; it changes path ranking. Not every language switch needs to be treated as equally suspicious, however, so the ranking model should leave room for language-pair-specific preferences rather than collapsing every mixed-language path into one flat bucket.

Useful local state is also slightly richer than "previous vocabulary language" alone. Even an early implementation benefits from knowing whether a boundary crosses passthrough fallback edges and from carrying coarse part-of-speech or connection information on adjacent dictionary-backed edges. Output-language bias remains an orthogonal ranking signal: edges whose lexemes belong to the target output language can receive a small bonus, since the user is more likely to want words native to the language they are writing in.

The prototype also distinguishes shared fallback unigram scoring from source-aware refinements. surfaceForm.frequency stays the shared cross-source unigram axis. Source-local metadata such as Mozc entry costs remain opt-in refinements layered on top of that shared axis rather than being treated as one global score scale.

The Japanese input method Mozc uses the same lattice + Viterbi approach, extended with bigram and connection costs that account for the plausibility of adjacent word pairs. Bibim can start with unigram costs and a previous-language state, then add coarse part-of-speech or bigram state later as the dictionary matures. In general the runtime is linear in the size of the expanded state graph, O(E ×ばつ S), where S is the number of retained states per node, not combinatorial in the total number of possible paths. User adaptation—boosting candidates the user selects frequently—is another natural extension. The prototype omits personalization; the production system can layer a per-user frequency adjustment on top of the base cost model.

Committed context across conversion boundaries

When the user commits a candidate (accepting a Viterbi path as final output), the last edge's metadata—vocabulary language, connection ID, and transition model—is preserved as a committed context object. This object is passed as an optional input to the next pipeline invocation. The transition cost between the committed context and the first edge of the new input is computed in the same way as any other adjacent-edge transition within a single lattice. The pipeline remains a pure function: the committed context is an explicit, immutable input parameter, not hidden session state.

Kangxi-centered normalization

Dictionary entries are stored in a canonical logographic form that is Kangxi-centered rather than Kangxi-exclusive. When a character has a Kangxi Dictionary form (康熙字典體), that form is used as the canonical form. Characters outside the Kangxi inventory, such as Korean-coined characters (國字) like 畓 (paddy field) or 乭 (a name element), are preserved in their source form and tagged as non-Kangxi canonicals.

This still gives the system a single fully-specified base form for output conversion. Japanese shinjitai merges several distinct traditional characters into one (e.g., 芸 ← 藝/芸); Chinese simplified does the same (e.g., 余 ← 餘/余). By normalizing to a Kangxi-centered canonical form, these mergers are recorded as forward mappings from a fully-specified base rather than as ambiguous reverse mappings.

Conversion from the canonical form to any target variant set is therefore a forward mapping from a fully-specified base. For most Kangxi-backed characters, this mapping is a deterministic, context-free, character-level lookup. Non-Kangxi canonicals pass through unchanged unless an explicit language-specific mapping is provided. A small number of canonical characters do map to different simplified Chinese forms depending on word context (e.g., 乾 → 干 in 乾燥 "dry" but 乾 in 乾坤 "heaven and earth"; 徵 → 征 in 徵兵 "conscription" but 徵 in 宮商角徵羽). These cases are handled by storing the correct output form in the dictionary entry, so disambiguation is resolved at dictionary-construction time rather than at runtime. The reverse direction (shinjitai → canonical form) would require word-level context to disambiguate at runtime; by choosing a Kangxi-centered canonical form, that problem is avoided entirely.

Output conversion

renderCandidate(
 path: CandidatePath,
 targetOutputVariant: OutputVariant,
) -> RenderedCandidate

The output stage consumes an annotated CandidatePath rather than a bare string, because it needs per-segment dictionary metadata. The targetOutputVariant is the global rendering policy for the whole candidate: it bundles what the earlier sections call output language together with its character-variant convention into one resolved target.

This distinction matters because rendering is driven by the target output policy, not by the source vocabulary language of each matched segment. A Korean lexeme normalized as 大學 may therefore render as 大学 when the chosen target output variant is Japanese shinjitai.

The final stage resolves each segment under the selected target output variant. Exact per-segment output forms stored in the dictionary take precedence, because lexicalized readings and context-dependent variant choices are properties of whole words or surface forms, not just isolated characters. Character-level conversion tables are the fallback mechanism for segments that do not carry a more specific override. For Japanese shinjitai, Chinese traditional, and other variants where most canonical characters map unambiguously, that fallback path is a deterministic, stateless character-by-character lookup.

Dictionary-backed segments are also checked against an explicit output-compatibility policy after rendering. Compatibility is judged on the rendered text itself, not on the source vocabulary language. This preserves useful cross-language cases such as Korean 大學 rendering as Japanese 大学, while still rejecting dictionary-backed results that leave incompatible script in the final output, such as hangul under a Japanese target or kana under a Korean target. In strict mode those incompatible dictionary-backed analyses are removed before candidate grouping. In penalize mode they remain available as fallback results, but incur a strong ranking penalty. Passthrough segments remain outside this policy for now.

Chinese simplified is mostly character-by-character, but a small set of canonical characters map to different simplified forms depending on word context (see Kangxi-centered normalization). For these characters, the pipeline reads the correct output form from the matched segment's dictionary entry rather than from the character-level table.

Non-CJK characters (kana, hangul syllables, Latin letters, punctuation, digits) pass through the output conversion stage unchanged.

Korean hangul output is slightly more involved. Most hanja can be replaced with their standard Korean reading character by character, but some words have conventional readings that diverge from strict per-character transliteration. 自轉車, for instance, is read as 자전거 in modern Korean, not 자전차. These cases require a lexeme-level reading stored in the dictionary rather than a mechanical character-by-character substitution.

For Korean mixed-script output, rendering preserves the segment boundaries of the same Viterbi path used in other modes. Each segment whose dictionary match is a Sino-Korean lexeme exposes both hanja and hangul renderings, with the hanja form as the default rendering (since the user has chosen mixed-script mode). The user can accept the hanja form or switch to the hangul reading, segment by segment. Non-Sino-Korean segments (native words, particles, loanwords) appear in hangul with no hanja alternative. This per-segment selection model follows the same pattern as phrase-level conversion in Japanese input methods.

Candidate objects

Each candidate shown to the user is not necessarily the same thing as one search path through the lattice. Internally, the search enumerates complete path analyses. The candidate-list stage then groups analyses that render to the same final output text, because distinct segmentations or lexical choices can collapse to one visible candidate.

A candidate object therefore carries both a user-facing rendering and the analysis metadata behind it:

• Final output text (after output conversion)
• Primary analysis (the best-ranked complete path for that rendered text)
• Alternative analyses that render to the same text
• Ordered segments for each analysis, each with its input span, matched surface form, underlying lexeme ID, canonical headword, vocabulary language, original reading, and any available morphology, per-segment cost, or rendering provenance
• Aggregate cost and score for each analysis, plus the ranking value used for the candidate itself
• User's actual input (e.g., 박물관)
• For mixed-script Korean, per-segment alternate renderings (hanja/hangul where applicable)

The UI layer can display as much or as little of this metadata as the platform allows. A desktop candidate window might show the segment path, source forms, and language tags together; a mobile keyboard might show only the output text, with details available on long press. The core pipeline produces the same rich objects everywhere; presentation is a platform concern.

Dictionary construction

The dictionary is the foundation. Without good dictionary data, the rest of the pipeline is academic.

Data sources

Korean hanja vocabulary is the natural starting point, since Korean hanja character forms closely follow the Kangxi Dictionary convention. Korean hanja dictionaries thus require minimal normalization into the canonical form. The main exceptions are Korean-coined characters (國字) such as 畓 (paddy field) or 乭 (a name element), which do not appear in the Kangxi Dictionary and are carried as source-preserved non-Kangxi canonicals.

Japanese vocabulary comes from open-source input method dictionaries, primarily Mozc. These are in shinjitai, so normalization into the canonical form is resolved and recorded during import. For the relatively small number of characters where shinjitai merges multiple Kangxi forms, disambiguation is needed during import (not at runtime). This can be done semi-automatically, using the Unicode Unihan database's kTraditionalVariant mappings to resolve most cases, with manual review for the remaining ambiguities.

Chinese vocabulary comes from open dictionaries such as CC-CEDICT. The same Unihan-assisted normalization applies.

The prototype now records this provenance explicitly. CC-CEDICT rows always use the unihan-traditional normalization policy. Mozc rows use unihan-traditional when the source headword contains kanji, but kana-only rows remain source-preserved. Standard Korean Dictionary rows preserve pure hanja headwords as korean-hanja-source-preserved, preserve lexicalized mixed-script spellings such as 애시當初 as mixed-script-source-preserved, and keep native Korean rows as source-preserved. This provenance is stored separately from the resulting canonical kind, because two rows may end up with similarly shaped canonical strings through different source policies.

The Unihan database itself provides the character-level scaffolding: readings in each language (kKoreanReading, kJapaneseOn, kJapaneseKun, kMandarin), variant mappings, and radical/stroke data.

Lexemes and surface forms

The dictionary distinguishes lexemes from surface forms. A lexeme is the normalized dictionary item (e.g., 行く, 博物館, 自轉車); a surface form is what can actually appear in the lattice (e.g., 行きましょう, 博物館, 자전거). The reading index maps transliterated strings to surface-form records, and each surface-form record points back to its lexeme, canonical headword, and output overrides. For the prototype, common inflected forms can be materialized during dictionary construction. A production system may later replace part of this with rule-based or finite-state morphology, but the rest of the pipeline need not change.

The roles of the main fields are intentionally distinct. canonicalHeadword is the normalization anchor used for dictionary merging and default character-level output conversion. sourceHeadword preserves the original dictionary spelling before normalization. surfaceText is the actual lattice-facing form that can be indexed and matched, whether it is a citation form, a materialized inflection, a lexicalized mixed-script spelling, or an independent functional segment such as Japanese ましょう. outputForms is reserved for exact whole-form overrides that cannot be reconstructed reliably from character-level mapping alone, such as 自轉車 -> 자전거. posDetail preserves the source dictionary's richer part-of-speech label separately from the coarse scoring bucket.

Storage format

For the prototype, a flat JSON or SQLite file is sufficient. Each entry preserves both its source form and the normalized canonical form, so that the original data can be traced and re-normalized if needed. The minimum fields are:

• Lexeme ID
• Canonical headword (Kangxi when available, otherwise a source-preserved non-Kangxi form)
• Canonical normalization policy provenance
• Source headword (the original form before canonical normalization)
• Surface form
• Language
• Reading(s)/indexed input keys
• Frequency (for Viterbi cost calculation)
• Coarse part-of-speech bucket (for fallback scoring heuristics)
• Source-language part-of-speech metadata (if available)
• Inflection class or other morphology features (if applicable)
• Optional connection class information (for future bigram/connection-cost modeling)
• Variant-set-specific output forms, where the default character-level mapping is context-dependent (see Output conversion)

Non-goals

Bibim is a CJK input method, not a translation engine. A few boundaries worth stating:

• Bibim does not perform machine translation. It resolves Chinese characters through shared Sino-vocabulary, not through semantic equivalence. If a word does not exist as a Sino-loanword in the target language, Bibim will not invent a correspondence.
• Bibim does not connect semantically similar words that share no logographic form. Korean 사람 and Japanese 人 mean roughly the same thing, but there is no shared character path between them.
• For vocabulary with no Sino-Chinese root (Japanese wago, Korean native words, loanwords from Western languages), Bibim offers phonetic transliteration but no cross-language lookup. The cross-language bridge exists only where the shared logographic layer exists.

Implementation strategy

Prototype: TypeScript web application

The prototype will be a browser-based application built in TypeScript. The core algorithms (lattice construction with transliteration, Viterbi search, output conversion) are pure logic with no platform dependencies, making TypeScript a natural fit for rapid iteration. The web UI needs only a text input field, a mode selector for the three layers, and a candidate popup.

Even during prototyping, the implementation should stay layered. The pure conversion engine should remain separate from language-specific CJK support such as transliteration tables and character-variant mappings; source-specific import adapters should remain separate again; checked-in prototype datasets should stay distinct from both; and the web app should be only a thin shell over those layers.

hangul jamo composition is handled by the browser's native Korean input during prototyping. The prototype concerns itself with what happens after the user has composed a hangul syllable or string: incremental lattice construction as each character arrives, prediction from prefix matches, candidate presentation, and partial commit with context carry-over.

The prototyping sequence:

1. Build a small seed dictionary (tens of entries, hand-curated) to validate the pipeline end-to-end.
2. Implement the pipeline stages against this seed data, including incremental lattice extension, trie-based prediction, and committed context across conversion boundaries.
3. Build a minimal web UI for interactive testing with live preedit updates, candidate selection, and partial commit.
4. Expand the dictionary from open-source input method data and Unihan.
5. Iterate on the cost model and segmentation quality.

A fixed prototype evaluation baseline now lives under packages/prototype-evaluation/baselines/. Run pnpm --filter @bibim/prototype-evaluation evaluate:baseline to refresh it. The command writes a machine-readable JSON artifact and a Hongdown-formatted Markdown summary. Optional local max source packages live under packages/web/public/data/. If those local packages are missing, the report keeps going and records the affected sections as skipped instead of failing outright. In the web demo, selecting a local max source package does not pull the whole package immediately; the demo first loads package metadata and then fetches only the lookup shards needed for the current input as typing continues.

The web demo is moving toward a thin-shell architecture. The browser UI will own only DOM concerns—focus, composition, input scheduling, URL state—and render snapshots received from an asynchronous conversion engine. Session reduction, candidate generation, input normalization, and lazy max-package shard loading will all run inside the engine. This keeps the prototype's shell-engine contract aligned with the production architecture's FFI boundary, so the same message shapes and ownership model carry over when the core is ported to Rust.

Production: Rust core with platform-specific shells

The production architecture targets multiple platforms: macOS and Linux (primary), iOS and Windows (secondary), Android (tertiary). The core logic will be implemented in Rust as a library of pure functions. Each platform will have a thin shell implementing the platform-specific input method or keyboard API (InputMethodKit on macOS, IBus/Fcitx on Linux, Text Services Framework on Windows, keyboard extensions on iOS).

The boundary between the Rust core and the platform shells is a typed, serializable async protocol. The shell sends requests (replace preedit, dispatch session command, initialize or reconfigure the engine) and the engine answers with declarative snapshots of the authoritative session state. The engine owns session reduction, candidate generation, input normalization, and resource loading; the shell owns platform-specific concerns such as focus, IME composition, and input scheduling. The protocol is defined as plain data types so the same message shapes cross the FFI boundary without async streams or callback interfaces.

The current TypeScript prototype now exercises this boundary explicitly across the packages/session/ and packages/prototype-runtime/ packages. These are not the final Rust interface, but they are the prototype artifacts that stabilize the session model and shell-neutral runtime layer before the port.

Candidate generation (lattice construction, Viterbi search, output conversion) runs synchronously per request against an in-memory dictionary snapshot, but the boundary itself is asynchronous: the engine can stream partial snapshots as optional dictionary resources load, and the shell applies only the latest snapshot, dropping stale responses from older requests. What requires asynchronous handling beyond lookup is the surrounding lifecycle: loading the dictionary from disk at startup, updating user learning data, syncing preferences across devices. Each platform's shell owns its own async runtime for these concerns and translates between the shell-to-engine protocol and the platform's input method API.

Why "Bibim"?

The name comes from the Korean word 비빔 (bibim), meaning "mixing," as in 비빔밥 (bibimbap, mixed rice). Bibim mixes phonetic readings, character forms, and vocabularies from across the CJK language family, finding the characters you need regardless of which language's pronunciation you start from.

It also happens to end in -im, which is a convenient echo of "input method."