charconv

A pure Crystal implementation of GNU libiconv. Converts text between 150+ character encodings using Unicode (UCS-4) as a pivot, with performance-first design.

• 150+ encodings: ASCII, UTF-8, UTF-16/32, ISO-8859-*, Windows codepages, Mac encodings, CJK (Shift_JIS, EUC-JP, GBK, Big5, EUC-KR, GB18030, ...), EBCDIC, and more
• Fast: 8-byte ASCII scanner with memcpy for ASCII-superset pairs, enum-based dispatch compiling to jump tables, table-driven single-byte codecs, zero allocations in the hot path
• Correct: Exhaustive byte-level tests against system iconv for every encoding
• Streaming: Buffer-based API for zero-copy conversion, plus IO wrapper for convenience
• GNU iconv compatible: Supports //IGNORE, //TRANSLIT, and combined flags

Add to your shard.yml:

dependencies:
 charconv:
 github: jackthorne/charconv

require "charconv"
# String/Bytes → Bytes
result = CharConv.convert("Hello, World!", "UTF-8", "ISO-8859-1")
result = CharConv.convert(input_bytes, "Shift_JIS", "UTF-8")
# With flags
result = CharConv.convert(input, "UTF-8", "ASCII//TRANSLIT") # transliterate
result = CharConv.convert(input, "UTF-8", "ASCII//IGNORE") # skip failures

converter = CharConv::Converter.new("EUC-JP", "UTF-8")
# You provide the buffers — zero allocations
src_consumed, dst_written = converter.convert(input_bytes, output_bytes)
# Call repeatedly until input is exhausted

File.open("input.txt", "r") do |input|
 File.open("output.txt", "w") do |output|
 CharConv.convert(input, output, "Shift_JIS", "UTF-8")
 end
end
# Or with a Converter instance for more control
converter = CharConv::Converter.new("GB18030", "UTF-8")
converter.convert(input_io, output_io, buffer_size: 16384)

Converter instances are not thread-safe — they hold mutable codec state for stateful encodings (ISO-2022-JP, UTF-7, HZ, etc.). Do not share a converter across fibers or threads. Instead, call #dup to create an independent copy:

converter = CharConv::Converter.new("ISO-2022-JP", "UTF-8")
# Each fiber gets its own copy with fresh state
10.times do
 spawn do
 my_conv = converter.dup
 my_conv.convert(input_bytes, output_bytes)
 end
end

charconv offers two error handling styles: status codes for streaming, and exceptions for one-shot conversion.

Status codes (streaming)

convert_with_status returns a ConvertStatus enum indicating why conversion stopped:

converter = CharConv::Converter.new("UTF-8", "ISO-8859-1")
src = input_bytes
dst = Bytes.new(src.size * 2)
consumed, written, status = converter.convert_with_status(src, dst)
case status
when .ok? then io.write(dst[0, written])
when .e2_big? then # grow buffer or flush and retry with src[consumed..]
when .eilseq? then raise "Invalid byte at position #{consumed}"
when .einval? then raise "Incomplete sequence at end of input"
end

Exceptions (one-shot)

The one-shot CharConv.convert raises CharConv::ConversionError on failure:

begin
 result = CharConv.convert(input, "UTF-8", "ISO-8859-1")
rescue ex : CharConv::ConversionError
 puts ex.message # e.g. "Conversion failed at byte 42 (42/100 bytes consumed)"
end

Using flags instead of error handling

//IGNORE silently skips invalid bytes and unencodable characters. //TRANSLIT attempts ASCII approximations (e.g. é → e, © → (c)). Combine both for maximum tolerance:

# Skip what can't be transliterated
result = CharConv.convert(input, "UTF-8", "ASCII//TRANSLIT//IGNORE")

When using convert or convert_with_status with your own buffers:

• Safe default: src.size * 4 covers worst-case expansion (1-byte source → 4-byte UTF-8)
• Exact ceiling: src.size * converter.to.max_bytes_per_char — never overestimates
• Same-family conversions (e.g. ISO-8859-1 → ISO-8859-2): output ≤ input, so src.size suffices
• If E2BIG is returned: double the buffer and retry, or flush written bytes and continue

converter = CharConv::Converter.new("EUC-JP", "UTF-8")
src = File.read("input.dat").to_slice
dst = Bytes.new(src.size * converter.to.max_bytes_per_char)
consumed, written, status = converter.convert_with_status(src, dst)

ISO-2022-JP, ISO-2022-CN, ISO-2022-KR, UTF-7, and HZ use escape sequences to switch between character sets. This means:

1. Call flush_encoder after the last chunk to emit any pending escape sequences (e.g. the switch-back-to-ASCII sequence in ISO-2022-JP):

   converter = CharConv::Converter.new("UTF-8", "ISO-2022-JP")
   consumed, written = converter.convert(src, dst)
   flush_len = converter.flush_encoder(dst, written)
   output.write(dst[0, written + flush_len])

2. Call reset before reusing the converter for a new document — otherwise the encoder assumes it's continuing the previous stream's state.

3. One-shot and IO methods handle this automatically — flush_encoder and reset are only needed when using the buffer-based streaming API directly.

CharConv.encoding_supported?("UTF-8") # => true
CharConv.encoding_supported?("NONEXISTENT") # => false
CharConv.list_encodings # => ["ASCII", "UTF-8", ...]

Unicode: ASCII, UTF-8, UTF-16BE/LE/BOM, UTF-32BE/LE/BOM, UCS-2, UCS-4, UTF-7, C99, Java

Western European: ISO-8859-1/15, CP1252, MacRoman, HP-ROMAN8, NEXTSTEP

Central/Eastern European: ISO-8859-2/3/4/10/13/14/16, CP1250, MacCentralEurope

Cyrillic: ISO-8859-5, CP1251, KOI8-R, KOI8-U, KOI8-RU, MacCyrillic, MacUkraine

Greek: ISO-8859-7, CP1253, MacGreek

Turkish: ISO-8859-9, CP1254, MacTurkish

Hebrew: ISO-8859-8, CP1255, MacHebrew

Arabic: ISO-8859-6, CP1256, MacArabic, CP864

Thai: ISO-8859-11, TIS-620, CP874, MacThai

Vietnamese: VISCII, TCVN, CP1258

Japanese: EUC-JP, Shift_JIS, CP932, ISO-2022-JP, ISO-2022-JP-1, ISO-2022-JP-2

Chinese (Simplified): GB2312, GBK, GB18030, EUC-CN, HZ, ISO-2022-CN

Chinese (Traditional): Big5, CP950, Big5-HKSCS, EUC-TW

Korean: EUC-KR, CP949, ISO-2022-KR, JOHAB

DOS/IBM: CP437, CP737, CP775, CP850, CP852, CP855, CP857, CP858, CP860-CP866, CP869

EBCDIC: CP037, CP273, CP277, CP278, CP280, CP284, CP285, CP297, CP423, CP424, CP500, CP905, CP1026

Other: ARMSCII-8, Georgian-Academy, Georgian-PS, PT154, KOI8-T, KZ-1048, MULELAO-1, ATARIST, RISCOS-LATIN1

charconv can transparently replace Crystal's libiconv dependency for all stdlib encoding operations (String#encode, String.new(bytes, encoding), IO#set_encoding).

require "charconv/stdlib"
# All stdlib encoding now uses charconv — no libiconv calls at runtime
"café".encode("ISO-8859-1")
String.new(bytes, "Shift_JIS")
io = File.open("data.txt")
io.set_encoding("EUC-JP")
io.gets_to_end # decoded through charconv

By default, libiconv is still linked but never called. To fully remove the libiconv dependency, compile with -Dwithout_iconv:

crystal build app.cr -Dwithout_iconv

charconv vs system libiconv, 1 MB input, --release mode.

Measured on Apple M3 Pro, Crystal 1.19.1, macOS. Run crystal spec spec/bench_spec.cr --release to reproduce.

Every conversion goes through a Unicode pivot:

Source bytes → UCS-4 codepoint → Target bytes
 decode() (pivot) encode()

For ASCII-superset encoding pairs (the vast majority), an 8-byte word scanner identifies ASCII runs and memcpys them directly, only falling back to the decode-pivot-encode loop for non-ASCII characters. This means ASCII-heavy text converts at memory bandwidth.

See ARCHITECTURE.md for the full design rationale.

By default, charconv embeds lookup tables for all 150+ encodings including CJK (~2 MB of table data). To build a smaller binary with only Western encodings:

crystal build app.cr -Dcharconv_minimal --release

The minimal set includes: ASCII, UTF-8, ISO-8859-1 through -16, CP1250-1258, KOI8-R/U, Mac encodings, DOS codepages, EBCDIC, UTF-16/32, UTF-7, C99, Java, and all other single-byte encodings. CJK encodings (Shift_JIS, EUC-JP, GBK, Big5, GB18030, EUC-KR, ISO-2022-*, HZ, etc.) are excluded — attempting to use them will return EILSEQ/ILUNI errors (the stubs reject all input).

crystal spec # run all tests
crystal spec spec/bench_spec.cr --release # run benchmarks

MIT