zg

zg provides Unicode text processing for Zig projects.

Unicode Version

The Unicode version supported by zg is 16.0.0.

Zig Version

The minimum Zig version required is 0.14.

Integrating zg into your Zig Project

You first need to add zg as a dependency in your build.zig.zon file. In your Zig project's root directory, run:

zig fetch --save https://codeberg.org/atman/zg/archive/v0.14.0-rc1.tar.gz

Then instantiate the dependency in your build.zig:

constzg=b.dependency("zg",.{});

A Modular Approach

zg is a modular library. This approach minimizes binary file size and memory requirements by only including the Unicode data required for the specified module. The following sections describe the various modules and their specific use case.

Init and Setup

The code examples will show the use of Module.init(allocator) to create the various modules. All of the allocating modules have a setup variant, which takes a pointer and allocates in-place.

Example use:

test"Setup form"{vargraphemes=tryallocator.create(Graphemes);deferallocator.destroy(graphemes);trygraphemes.setup(allocator);defergraphemes.deinit(allocator);}

Code Points

In the code_point module, you'll find a data structure representing a single code point, CodePoint, and an Iterator to iterate over the code points in a string.

In your build.zig:

exe.root_module.addImport("code_point",zg.module("code_point"));

In your code:

constcode_point=@import("code_point");test"Code point iterator"{conststr="Hi 😊";variter=code_point.Iterator{.bytes=str};vari:usize=0;while(iter.next())|cp|:(i+=1){// The `code` field is the actual code point scalar as a `u21`.if(i==0)tryexpect(cp.code=='H');if(i==1)tryexpect(cp.code=='i');if(i==2)tryexpect(cp.code==' ');if(i==3){tryexpect(cp.code=='😊');// The `offset` field is the byte offset in the// source string.tryexpect(cp.offset==3);tryexpectEqual(code_point.CodePoint,code_point.decodeAtIndex(str,cp.offset));// The `len` field is the length in bytes of the// code point in the source string.tryexpect(cp.len==4);}}}

Grapheme Clusters

Many characters are composed from more than one code point. These are known as Grapheme Clusters, and the Graphemes module has a data structure to represent them, Grapheme, and an Iterator to iterate over them in a string.

In your build.zig:

exe.root_module.addImport("Graphemes",zg.module("Graphemes"));

In your code:

constGraphemes=@import("Graphemes");test"Grapheme cluster iterator"{constgraph=tryGraphemes.init(allocator);defergraph.deinit(allocator);conststr="He\u{301}";// Hévariter=graph.iterator(str);vari:usize=0;while(iter.next())|gc|:(i+=1){// The `len` field is the length in bytes of the// grapheme cluster in the source string.if(i==0)tryexpect(gc.len==1);if(i==1){tryexpect(gc.len==3);// The `offset` in bytes of the grapheme cluster// in the source string.tryexpect(gc.offset==1);// The `bytes` method returns the slice of bytes// that comprise this grapheme cluster in the// source string `str`.tryexpectEqualStrings("e\u{301}",gc.bytes(str));}}}

Unicode General Categories

To detect the general category for a code point, use the GeneralCategories module.

In your build.zig:

exe.root_module.addImport("GeneralCategories",zg.module("GeneralCategories"));

In your code:

constGeneralCategories=@import("GeneralCategories");test"General Categories"{constgen_cat=tryGeneralCategories.init(allocator);defergen_cat.deinit(allocator);// The `gc` method returns the abbreviated General Category.// These abbreviations and descriptive comments can be found// in the source file `src/GenCatData.zig` as en enum.tryexpect(gen_cat.gc('A')==.Lu);// Lu: uppercase lettertryexpect(gen_cat.gc('3')==.Nd);// Nd: decimal number// The following are convenience methods for groups of General// Categories. For example, all letter categories start with `L`:// Lu, Ll, Lt, Lo.tryexpect(gen_cat.isControl(0));tryexpect(gen_cat.isLetter('z'));tryexpect(gen_cat.isMark('\u{301}'));tryexpect(gen_cat.isNumber('3'));tryexpect(gen_cat.isPunctuation('['));tryexpect(gen_cat.isSeparator(' '));tryexpect(gen_cat.isSymbol('©'));}

Unicode Properties

You can detect common properties of a code point with the Properties module.

In your build.zig:

exe.root_module.addImport("Properties",zg.module("Properties"));

In your code:

constProperties=@import("Properties");test"Properties"{constprops=tryProperties.init(allocator);deferprops.deinit(allocator);// Mathematical symbols and letters.tryexpect(props.isMath('+'));// Alphabetic only code points.tryexpect(props.isAlphabetic('Z'));// Space, tab, and other separators.tryexpect(props.isWhitespace(' '));// Hexadecimal digits and variations thereof.tryexpect(props.isHexDigit('f'));tryexpect(!props.isHexDigit('z'));// Accents, dieresis, and other combining marks.tryexpect(props.isDiacritic('\u{301}'));// Unicode has a specification for valid identifiers like// the ones used in programming and regular expressions.tryexpect(props.isIdStart('Z'));// Identifier start charactertryexpect(!props.isIdStart('1'));tryexpect(props.isIdContinue('1'));// The `X` versions add some code points that can appear after// normalizing a string.tryexpect(props.isXidStart('\u{b33}'));// Extended identifier start charactertryexpect(props.isXidContinue('\u{e33}'));tryexpect(!props.isXidStart('1'));// Note surprising Unicode numeric type properties!tryexpect(props.isNumeric('\u{277f}'));tryexpect(!props.isNumeric('3'));// 3 is not numeric!tryexpect(props.isDigit('\u{2070}'));tryexpect(!props.isDigit('3'));// 3 is not a digit!tryexpect(props.isDecimal('3'));// 3 is a decimal digit}

Letter Case Detection and Conversion

To detect and convert to and from different letter cases, use the LetterCasing module.

In your build.zig:

exe.root_module.addImport("LetterCasing",zg.module("LetterCasing"));

In your code:

constLetterCasing=@import("LetterCasing");test"LetterCasing"{constcase=tryLetterCasing.init(allocator);defercase.deinit(allocator);// Upper and lower case.tryexpect(case.isUpper('A'));tryexpect('A'==case.toUpper('a'));tryexpect(case.isLower('a'));tryexpect('a'==case.toLower('A'));// Code points that have case.tryexpect(case.isCased('É'));tryexpect(!case.isCased('3'));// Case detection and conversion for strings.tryexpect(case.isUpperStr("HELLO 123!"));constucased=trycase.toUpperStr(allocator,"hello 123");deferallocator.free(ucased);tryexpectEqualStrings("HELLO 123",ucased);tryexpect(case.isLowerStr("hello 123!"));constlcased=trycase.toLowerStr(allocator,"HELLO 123");deferallocator.free(lcased);tryexpectEqualStrings("hello 123",lcased);}

Normalization

Unicode normalization is the process of converting a string into a uniform representation that can guarantee a known structure by following a strict set of rules. There are four normalization forms:

Canonical Composition (NFC)

The most compact representation obtained by first decomposing to Canonical Decomposition and then composing to NFC.

Compatibility Composition (NFKC)

The most comprehensive composition obtained by first decomposing to Compatibility Decomposition and then composing to NFKC.

Canonical Decomposition (NFD)

Only code points with canonical decompositions are decomposed. This is a more compact and faster decomposition but will not provide the most comprehensive normalization possible.

Compatibility Decomposition (NFKD)

The most comprehensive decomposition method where both canonical and compatibility decompositions are performed recursively.

zg has methods to produce all four normalization forms in the Normalize module.

In your build.zig:

exe.root_module.addImport("Normalize",zg.module("Normalize"));

In your code:

constNormalize=@import("Normalize");test"Normalize"{constnormalize=tryNormalize.init(allocator);defernormalize.deinit(allocator);// NFC: Canonical compositionconstnfc_result=trynormalize.nfc(allocator,"Complex char: \u{3D2}\u{301}");defernfc_result.deinit(allocator);tryexpectEqualStrings("Complex char: \u{3D3}",nfc_result.slice);// NFKC: Compatibility compositionconstnfkc_result=trynormalize.nfkc(allocator,"Complex char: \u{03A5}\u{0301}");defernfkc_result.deinit(allocator);tryexpectEqualStrings("Complex char: \u{038E}",nfkc_result.slice);// NFD: Canonical decompositionconstnfd_result=trynormalize.nfd(allocator,"Héllo World! \u{3d3}");defernfd_result.deinit(allocator);tryexpectEqualStrings("He\u{301}llo World! \u{3d2}\u{301}",nfd_result.slice);// NFKD: Compatibility decompositionconstnfkd_result=trynormalize.nfkd(allocator,"Héllo World! \u{3d3}");defernfkd_result.deinit(allocator);tryexpectEqualStrings("He\u{301}llo World! \u{3a5}\u{301}",nfkd_result.slice);// Test for equality of two strings after normalizing to NFC.tryexpect(trynormalize.eql(allocator,"foé","foe\u{0301}"));tryexpect(trynormalize.eql(allocator,"foΎ","fo\u{03D2}\u{0301}"));}

The Result returned by normalization functions may or may not be copied from the inputs given. For example, an all-ASCII input does not need to be a copy, and will be a view of the original slice. Calling result.deinit(allocator) will only free an allocated Result, not one which is a view. Thus it is safe to do unconditionally.

This does mean that the validity of a Result can depend on the original string staying in memory. To ensure that your Result is always a copy, you may call try result.toOwned(allocator), which will only make a copy if one was not already made.

Caseless Matching via Case Folding

Unicode provides a more efficient way of comparing strings while ignoring letter case differences: case folding. When you case fold a string, it's converted into a normalized case form suitable for efficient matching. Use the CaseFold module for this.

In your build.zig:

exe.root_module.addImport("CaseFolding",zg.module("CaseFolding"));

In your code:

constCaseFolding=@import("CaseFolding");test"Caseless matching"{// We need Unicode case fold data.constcase_fold=tryCaseFolding.init(allocator);defercase_fold.deinit(allocator);// `compatCaselessMatch` provides the deepest level of caseless// matching because it decomposes fully to NFKD.consta="Héllo World! \u{3d3}";constb="He\u{301}llo World! \u{3a5}\u{301}";tryexpect(trycase_fold.compatCaselessMatch(allocator,a,b));constc="He\u{301}llo World! \u{3d2}\u{301}";tryexpect(trycase_fold.compatCaselessMatch(allocator,a,c));// `canonCaselessMatch` isn't as comprehensive as `compatCaselessMatch`// because it only decomposes to NFD. Naturally, it's faster because of this.tryexpect(!trycase_fold.canonCaselessMatch(allocator,a,b));tryexpect(trycase_fold.canonCaselessMatch(allocator,a,c));}

Case folding needs to use the Normalize module in order to produce the compatibility forms for comparison. If you are already using a Normalize for other purposes, CaseFolding can borrow it:

constCaseFolding=@import("CaseFolding");constNormalize=@import("Normalize");test"Initialize With a Normalize"{constnormalize=tryNormalize.init(allocator);// You're responsible for freeing this:defernormalize.deinit(allocator);constcase_fold=tryCaseFolding.initWithNormalize(allocator,normalize);// This will not free your normalize when it runs first.defercase_fold.deinit(allocator);}

This has a setupWithNormalize variant as well, note that this also takes a Normalize struct, and not a pointer to it.

Display Width of Characters and Strings

When displaying text with a fixed-width font on a terminal screen, it's very important to know exactly how many columns or cells each character should take. Most characters will use one column, but there are many, like emoji and East- Asian ideographs that need more space. The DisplayWidth module provides methods for this purpose. It also has methods that use the display width calculation to center, padLeft, padRight, and wrap text.

In your build.zig:

exe.root_module.addImport("DisplayWidth",zg.module("DisplayWidth"));

In your code:

constDisplayWidth=@import("DisplayWidth");test"Display width"{constdw=tryDisplayWidth.init(allocator);deferdw.deinit(allocator);// String display widthtryexpectEqual(@as(usize,5),dw.strWidth("Hello\r\n"));tryexpectEqual(@as(usize,8),dw.strWidth("Hello 😊"));tryexpectEqual(@as(usize,8),dw.strWidth("Héllo 😊"));tryexpectEqual(@as(usize,9),dw.strWidth("Ẓ̌á̲l͔̝̞̄̑͌g̖̘̘̔̔͢͞͝o̪̔T̢̙̫̈̍͞e̬͈͕͌̏͑x̺̍ṭ̓̓ͅ"));tryexpectEqual(@as(usize,17),dw.strWidth("슬라바 우크라이나"));// Centering textconstcentered=trydw.center(allocator,"w😊w",10,"-");deferallocator.free(centered);tryexpectEqualStrings("---w😊w---",centered);// Pad leftconstright_aligned=trydw.padLeft(allocator,"abc",9,"*");deferallocator.free(right_aligned);tryexpectEqualStrings("******abc",right_aligned);// Pad rightconstleft_aligned=trydw.padRight(allocator,"abc",9,"*");deferallocator.free(left_aligned);tryexpectEqualStrings("abc******",left_aligned);// Wrap textconstinput="The quick brown fox\r\njumped over the lazy dog!";constwrapped=trydw.wrap(allocator,input,10,3);deferallocator.free(wrapped);constwant=\\The quick\\brown fox\\jumped\\over the\\lazy dog!;tryexpectEqualStrings(want,wrapped);}

This module has build options. The first is cjk, which will consider ambiguous characters as double-width.

To choose this option, add it to the dependency like so:

constzg=b.dependency("zg",.{.cjk=true,});

The other options are c0_width and c1_width. The standard behavior is to treat C0 and C1 control codes as zero-width, except for delete and backspace, which are -1 (the logic ensures that a strWidth is always at least 0). If printing control codes with replacement characters, it's necessary to assign these a width, hence the options. When provided these values must fit in an i4, this allows for C1s to be printed as \u{80} if desired.

DisplayWidth uses the Graphemes module internally. If you already have one, it can be borrowed using DisplayWidth.initWithGraphemes(allocator, graphemes) in the same fashion as shown for CaseFolding and Normalize.

Scripts

Unicode categorizes code points by the Script in which they belong. A Script collects letters and other symbols that belong to a particular writing system. You can detect the Script for a code point with the Scripts module.

In your build.zig:

exe.root_module.addImport("Scripts",zg.module("Scripts"));

In your code:

constScripts=@import("Scripts");test"Scripts"{constscripts=tryScripts.init(allocator);deferscripts.deinit(allocator);// To see the full list of Scripts, look at the// `src/Scripts.zig` file. They are list in an enum.tryexpect(scripts.script('A')==.Latin);tryexpect(scripts.script('Ω')==.Greek);tryexpect(scripts.script('צ')==.Hebrew);}

Relation to Ziglyph

zg is a total re-write of some of the components of Ziglyph. The idea was to reduce binary size and improve performance. These goals were achieved by using trie-like data structures (inspired by Ghostty's implementation) instead of generated functions. Where Ziglyph uses a function call, zg uses an array lookup, which is quite faster. In addition, all these data structures in zg are loaded at runtime from compressed versions in the binary. This allows for smaller binary sizes at the expense of increased memory footprint at runtime.

Benchmarks demonstrate the above stated goals have been met:

Binary sizes =======
172K ziglyph_case
109K zg_case
299K ziglyph_caseless
175K zg_caseless
91K ziglyph_codepoint
91K zg_codepoint
108K ziglyph_grapheme
109K zg_grapheme
208K ziglyph_normalizer
175K zg_normalize
124K ziglyph_width
109K zg_width
Benchmarks ==========
Ziglyph toUpperStr/toLowerStr: result: 7756580, took: 74
Ziglyph isUpperStr/isLowerStr: result: 110959, took: 17
zg toUpperStr/toLowerStr: result: 7756580, took: 58
zg isUpperStr/isLowerStr: result: 110959, took: 11
Ziglyph Normalizer.eqlCaseless: result: 626, took: 479
zg CaseFolding.canonCaselessMatch: result: 626, took: 296
zg CaseFolding.compatCaselessMatch: result: 626, took: 604
Ziglyph CodePointIterator: result: 3691806, took: 2.5
zg code_point.Iterator: result: 3691806, took: 3.3
Ziglyph GraphemeIterator: result: 3691806, took: 78
zg Graphemes.Iterator: result: 3691806, took: 31
Ziglyph Normalizer.nfkc: result: 3856654, took: 411
zg Normalize.nfkc: result: 3856654, took: 208
Ziglyph Normalizer.nfc: result: 3878290, took: 56
zg Normalize.nfc: result: 3878290, took: 31
Ziglyph Normalizer.nfkd: result: 3928890, took: 163
zg Normalize.nfkd: result: 3928890, took: 101
Ziglyph Normalizer.nfd: result: 3950526, took: 160
zg Normalize.nfd: result: 3950526, took: 101
Ziglyph Normalizer.eql: result: 626, took: 321
Zg Normalize.eql: result: 626, took: 60
Ziglyph display_width.strWidth: result: 3700914, took: 89
zg DisplayWidth.strWidth: result: 3700914, took: 46

These results were obtained on a MacBook Pro (2021) with M1 Pro and 16 GiB of RAM.

In contrast to Ziglyph, zg does not have:

• Word segmentation
• Sentence segmentation
• Collation

It's possible that any missing functionality will be added in future versions, but only if enough demand is present in the community.