[search] Use ICU for CJK word break #9

eisa01 commented 2026年03月15日 21:13:58 +01:00

Done fully by Claude Opus 4.6. Had lots of trouble compiling the new .dat file, but managed to debug it

Summary

• Enable ICU BreakIterator with CJK dictionary support to fix search for partial Chinese/Japanese/Korean place names (issue #3835)
• Searching "博物館" now finds "國立臺灣博物館"; searching "博物" does not — correct word-level precision

Changes

ICU build system

• 3party/icu/uconfig_local.h: Removed UCONFIG_NO_BREAK_ITERATION and UCONFIG_NO_FILTERED_BREAK_ITERATION
• 3party/icu/CMakeLists.txt: Added 13 break iterator source files to icuuc (rbbi*.cpp, brkiter.cpp, ubrk.cpp, dictbe.cpp, dictionarydata.cpp)
• 3party/icu/icu_filter.json: Added brkitr_rules and brkitr_dictionaries features

ICU data

• data/icudt75l.dat: Rebuilt with break iterator rules and CJK dictionary — grows from 1.3 MB to 4.5 MB. New contents include word.brk, cjdict.dict (Chinese/Japanese word list, BSD-licensed via Chromium/Libtabe/IPADIC), and the full set of break rules.

Search tokenization

• libs/indexer/search_string_utils.hpp: Added IsCJKChar() range check and SegmentCJKWords() declaration; modified SplitUniString to detect CJK runs and delegate to SegmentCJKWords instead of treating them as a single token
• libs/indexer/search_string_utils.cpp: Implemented SegmentCJKWords() using icu::BreakIterator::createWordInstance("zh"), which activates the CJK dictionary for word-boundary detection; falls back to per-character tokenization if ICU data is unavailable

Tests

• libs/search/search_integration_tests/processor_test.cpp: Added ChinesePOISearchTest covering full-name, suffix-word, and multi-word queries against a Chinese POI

How to rebuild icudt75l.dat

The data rebuild requires a bootstrap step (ICU tools need Unicode property data to compile break rules):

cd 3party/icu && mkdir -p build && cd build
ICU_DATA_FILTER_FILE=../icu_filter.json \
 ../icu/icu4c/source/configure --disable-shared --enable-static \
 --disable-renaming --disable-extras --disable-icuio \
 --disable-tests --disable-samples --with-data-packaging=archive
make -j$(nproc)
# Bootstrap before running make (needed once per clean build):
cd data
DYLD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$DYLD_LIBRARY_PATH \
 ../bin/gencnval -d ./out/build/icudt75l \
 -s ../../icu/icu4c/source/data/mappings convrtrs.txt
cp ../../icu/icu4c/source/data/in/*.icu \
 ../../icu/icu4c/source/data/in/*.nrm \
 ./out/build/icudt75l/
make -j$(nproc)
cp out/icudt75l.dat ../../../../data/

Done fully by Claude Opus 4.6. Had lots of trouble compiling the new .dat file, but managed to debug it > ## Summary > > - Enable ICU `BreakIterator` with CJK dictionary support to fix search for partial Chinese/Japanese/Korean place names (issue #3835) > - Searching "博物館" now finds "國立臺灣博物館"; searching "博物" does not — correct word-level precision > > ## Changes > > ### ICU build system > - **`3party/icu/uconfig_local.h`**: Removed `UCONFIG_NO_BREAK_ITERATION` and `UCONFIG_NO_FILTERED_BREAK_ITERATION` > - **`3party/icu/CMakeLists.txt`**: Added 13 break iterator source files to `icuuc` (`rbbi*.cpp`, `brkiter.cpp`, `ubrk.cpp`, `dictbe.cpp`, `dictionarydata.cpp`) > - **`3party/icu/icu_filter.json`**: Added `brkitr_rules` and `brkitr_dictionaries` features > > ### ICU data > - **`data/icudt75l.dat`**: Rebuilt with break iterator rules and CJK dictionary — grows from 1.3 MB to 4.5 MB. New contents include `word.brk`, `cjdict.dict` (Chinese/Japanese word list, BSD-licensed via Chromium/Libtabe/IPADIC), and the full set of break rules. > > ### Search tokenization > - **`libs/indexer/search_string_utils.hpp`**: Added `IsCJKChar()` range check and `SegmentCJKWords()` declaration; modified `SplitUniString` to detect CJK runs and delegate to `SegmentCJKWords` instead of treating them as a single token > - **`libs/indexer/search_string_utils.cpp`**: Implemented `SegmentCJKWords()` using `icu::BreakIterator::createWordInstance("zh")`, which activates the CJK dictionary for word-boundary detection; falls back to per-character tokenization if ICU data is unavailable > > ### Tests > - **`libs/search/search_integration_tests/processor_test.cpp`**: Added `ChinesePOISearchTest` covering full-name, suffix-word, and multi-word queries against a Chinese POI > > ## How to rebuild `icudt75l.dat` > > The data rebuild requires a bootstrap step (ICU tools need Unicode property data to compile break rules): > > ```bash > cd 3party/icu && mkdir -p build && cd build > ICU_DATA_FILTER_FILE=../icu_filter.json \ > ../icu/icu4c/source/configure --disable-shared --enable-static \ > --disable-renaming --disable-extras --disable-icuio \ > --disable-tests --disable-samples --with-data-packaging=archive > make -j$(nproc) > > # Bootstrap before running make (needed once per clean build): > cd data > DYLD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$DYLD_LIBRARY_PATH \ > ../bin/gencnval -d ./out/build/icudt75l \ > -s ../../icu/icu4c/source/data/mappings convrtrs.txt > cp ../../icu/icu4c/source/data/in/*.icu \ > ../../icu/icu4c/source/data/in/*.nrm \ > ./out/build/icudt75l/ > make -j$(nproc) > cp out/icudt75l.dat ../../../../data/ >

This pull request can be merged automatically.

You are not authorized to merge this pull request.