- Emacs Lisp 100%
|
|
||
|---|---|---|
| Okapi-defaults | add srx files + readme | |
| OmegaT-defaults | add srx files + readme | |
| readme.org | read me typo and cull snark | |
| sentex-convert.el | rename repo | |
| sentex-icu4j-rules-converted.el | rename repo | |
| sentex-okapi-alt-rules-converted.el | rename repo | |
| sentex-omegat-rules-converted.el | rename repo | |
| sentex-regexes.el | rename repo | |
| sentex-unicode-properties-used-by-ICU-regexes.txt | rename repo | |
| sentex.el | FIXME for looking-back-forward-map | |
| test-texts.txt | add a sentence to the test text | |
sentex
This repo ports complex sentence-ending rules from OmegaT and the Okapi Framework to Emacs Lisp, and then provides sentence-based functions based on the rules. Two movement functions exist: sentex-forward-sentence and sentex-backward-sentence, as well as sentex-kill-sentence.
To set the framework to use, customize sentex-ruleset-framework. To set the language ruleset to use, customize sentex-current-language, or if you want it to be buffer-local, run sentex-set-language-for-buffer. OmegaT and the Okapi Framework each support different languages, so if your language doesn't appear in the completion list, try using a different framework. Languages supported by the currently set framework are returned by sentex-get-valid-langs.
As this is new code, it hasn't been tested much. In particular, I don't yet know how it interacts with custom sentence-end values, or other possible snags, such as anyone else's Emacs other than mine (or even mine).
conversion
OmegaT and the Okapi Framework use ICU regular expressions stored in SRX files (copied into this repo) to create sophisticated rules for segmenting texts. Each rule has a before break and after break regular expression, as well as a boolean break or don't break value.
The rules have been converted in part by hand (the Unicode property parts of the ICU regexes) and partly using pcre2el (for the rest). The process is ad-hoc, there may be errors, they'll be corrected as they're found.
The bulk converted ruleset files used by sentex have the suffix -rules-converted.el. There are three kinds: OmegaT, Okapi alt, and ICU4J.
custom rules
You can add your own custom rules by customizing sentex-custom-rules-regex-list. See its default value for the form the rules take.
features
- sophisticated non-break and break rules for common abbreviations and other situations where a dot doesn't end a sentence.
- add your own custom rules
- doesn't assume sentences begin with a capital letter.
- should work with filled text.
- obviates the need to use two spaces for periods, which is not a solution if you have to work with texts that you are not authoring yourself.
contributing
Contributions are most welcome.
If your regex foo is half-decent, it's already better than mine.