disarm

PyPI Crates.io Documentation License: MIT

Identify malicious attacks hiding in text.

раypal.com — Cyrillic а (U+0430) and р (U+0440) — renders identically to paypal.com and is a different string. disarm finds that substitution and folds it back to its Unicode TR39 prototype, strips bidi overrides, zero-width and control characters, and flags spoofed hostnames — the Unicode layer your validation, dedup, moderation and logging code is missing.

One pure-Rust core, with bindings for Python, Rust, Ruby, Node.js, Java/Kotlin and C.

from disarm import canonicalize, is_suspicious_hostname
# U+202E is a right-to-left override and U+200B a zero-width space. Neither is
# visible, and both survive a copy-paste straight into your database.
assert canonicalize("\u202eexample\u200b.com") == "example.com"
# U+0397 is Greek capital eta and U+13D4 Cherokee letter wa. They render as H and W.
assert canonicalize("\u0397ello \u13d4orld") == "Hello World"
# Cyrillic small a (U+0430) standing in for Latin a: renders as "apple.com".
suspicious, analysis = is_suspicious_hostname("\u0430pple.com")
assert suspicious and analysis.canonical == "apple.com"

Try it in your browser · Documentation · API reference

pip install disarm # Python 3.10+ (wheels for Linux, macOS, Windows)
cargo add disarm # Rust 1.81+ (pure Rust — no Python, no pyo3)
npm install disarm # Node.js 14+
gem install disarm # Ruby 3.1+

• Confusable folding — TR39 visual mapping, plus what it misses
• Obfuscation stripping — bidi controls, zero-width characters, zalgo, emoji
• Hostname / IDN analysis — mixed-script, whole-script and bidi checks
• Ready-made pipelines — canonicalize, catalog_key, search_key, LLM and RAG profiles
• Transliteration — BGN/PCGN, ISO 9 and GOST; 83 language profiles
• Normalization, slugs & filenames — case folding, graphemes, encoding detection

from disarm import canonicalize, collapse_whitespace, slugify, strip_obfuscation, transliterate
# Cyrillic er (U+0440) and es (U+0441) folded to Latin p and c — visual (TR39) mapping.
assert strip_obfuscation("\u0440rodu\u0441t") == "product"
# No-break space (U+00A0), ideographic space (U+3000), thin space (U+2009) and a
# line separator (U+2028) all collapse to one plain ASCII space.
assert collapse_whitespace("Ada\u00a0\u3000Lovelace\u2009\u2028King") == "Ada Lovelace King"
# Their zero-width look-alikes are not whitespace at all — U+200B and U+FEFF are
# format characters, so neither str.split() nor collapse_whitespace touches them.
assert collapse_whitespace("A\u200bB\ufeffC") == "A\u200bB\ufeffC"
assert canonicalize("A\u200bB\ufeffC") == "ABC"
# Phonetic romanization: a different mapping, and not a defence.
assert transliterate("Київ", lang="uk") == "Kyiv"
assert slugify("Héllo Wörld") == "hello-world"

Does it work? On the XMR confusable-recovery metric, disarm's visual mapping scores 0.63–0.68, against ≤ 0.19 for phonetic transliterators (unidecode, anyascii, uroman) and 0.10 for NFKC. → the evidence · what it misses

What does it cost? ~450M chars/sec on Latin (×ばつ Unidecode), ~106M on Cyrillic, ~712K slugs/sec (×ばつ python-slugify), ~65 ns for an already-ASCII call. Hardware-dependent and directional, not guarantees. → full results · how to read them · where disarm is slower

Both come from "Fire Extinguishers Full of Gasoline": 435,864 observations over eight tools, six attack types, three tasks and two model architectures. Zenodo · CITATION.cff

Each binding reads like its own ecosystem — snake_case in Ruby, camelCase and .d.ts in Node, builders in Java — over one shared core, so every language returns the same answer.

Wheels, gems and addons are precompiled — no local Rust toolchain needed. The core crate is unsafe_code = "forbid" and stays pure Rust; BINDINGS.md is the bar a new binding has to meet.

• Defense in depth, not a complete control. disarm folds the confusables it bundles and strips the format characters it enumerates. The confusable space is larger than any table, so measure your residue with unmapped_confusables() rather than inferring it. Threat model.
• Not an output sanitizer. disarm normalizes input. It performs no escaping — <script>alert(1)</script> passes through unchanged, and NFKC can even surface ASCII metacharacters from fullwidth look-alikes. Keep encoding at the output sink (framework auto-escaping, DOMPurify, parameterized queries); run disarm before it.
• transliterate() is not a security control. It romanizes phonetically. For homoglyph defense use normalize_confusables() / strip_obfuscation().

CONFUSABLES_VERSION reports which confusables.txt release the bundled tables were folded from, so a deployment can answer "am I stale?" without inferring it from behaviour (provenance).

Found a bypass? Report it under the security policy rather than in a public issue.

MIT