Description
Currently, the search functionality in Helix Note does not handle Chinese text properly. It appears that sentences are being treated as single, continuous tokens (words) because Chinese does not use spaces between characters.
Steps to Reproduce
Create a note containing a Chinese sentence, for example: 这是一段测试文字 (This is a test sentence).
Attempt to search for a keyword in the middle of the sentence, such as 测试 (test).
Observe that the search fails to return the note.
Search for the very first character 这 (this), and the note will likely appear.
Expected Behavior
Search should be able to identify and match substrings within a Chinese sentence, regardless of whether the keyword starts at the beginning of the string.
Actual Behavior
The search only seems to work if the query matches the start of the "word." Since the entire sentence is identified as one long word due to the lack of spaces, mid-sentence searching is effectively broken.
Potential Cause
This is likely a tokenization issue. Many search engines or indexing libraries default to space-based delimiters. For Chinese, a character-based or N-gram tokenization approach is required to allow searching for characters or phrases within a continuous string of text.
Suggested Fix / Workaround
Implement a specialized tokenizer for CJK (Chinese, Japanese, Korean) characters.
Enable N-gram indexing to allow for partial matching within tokens.
Update the regex or search logic to treat individual CJK characters as searchable units.