Jun 29, 2012 by Tom Christiansen
This series has shown you several features of Unicode by example, as well as several techniques for working with Unicode correctly and easily with recent releases of Perl 5. By now you know more than many programmers do about Unicode,…
Jun 22, 2012 by Tom Christiansen
℞ 44: PROGRAM: Demo of Unicode collation and printing The past several weeks of Unicode recipes have explained how Unicode works and shown how to use it in your programs. If you’ve gone through those recipes, you now understand more…
Jun 20, 2012 by Tom Christiansen
℞ 43: Unicode text in DBM hashes, the easy way Some Perl libraries require you to jump through hoops to handle Unicode data. Would that everything worked as easily as Perl’s open pragma! For DBM files, here’s how to implicitly…
Jun 18, 2012 by Tom Christiansen
℞ 42: Unicode text in DBM hashes, the tedious way While Perl 5 has long been very careful about handling Unicode correctly inside the world of Perl itself, every time you leave the Perl internals, you cross a boundary at…
Jun 12, 2012 by Tom Christiansen
℞ 41: Unicode linebreaking If you’ve ever tried to fit a large amount of text into a display area too narrow for the full width of the text, you’ve dealt with the joy of linebreaking (or word wrapping). As you…
Jun 11, 2012 by Tom Christiansen
℞ 40: Case- and accent-insensitive locale comparisons You now know how to compare Unicode strings while ignoring case and accent differences. This approach uses the standard Unicode collation algorithm. To perform a similar comparison while respecting a specific locale’s rules,…
Jun 8, 2012 by Tom Christiansen
℞ 39: Case- and accent-insensitive comparisons As you’ve noticed by now, many Unicode strings have multiple possible representations. Comparing two Unicode strings for equality requires far more than merely comparing their codepoints. Not only must you account for multiple representations,…
Jun 7, 2012 by Tom Christiansen
℞ 38: Making cmp work on text instead of codepoints Even with Perl 5.12’s “unicode_strings” feature, some of Perl’s core operations do not perform as expected on Unicode strings by default. For example, how is the cmp operator to know…
Jun 5, 2012 by Tom Christiansen
℞ 37: Unicode locale collation As you’ve already seen, Unicode-aware sorting respects Unicode character properties. You can’t sort by codepoint and expect to get accurate results, not even if you stick with pure ASCII. The world is a complicated place….
Jun 4, 2012 by Tom Christiansen
℞ 36: Case- and accent-insensitive Unicode sort The Unicode Collation Algorithm defines several levels of collation strength by which you can specify certain character properties as relevant or irrelevant to the collation ordering. In simple terms, you can use collation…
Jun 1, 2012 by Tom Christiansen
℞ 35: Unicode collation Sorting—even pure ASCII—seems easy, at least if you know the alphabet song. Yet even something this simple gets complicated if you sort merely by codepoint. You get numbers coming in the midst of letters. You get…
May 31, 2012 by Tom Christiansen
℞ 34: Unicode column-width for printing Perl’s printf, sprintf, and format think all codepoints take up 1 print column, but many codepoints take 0 or 2. If you use any of these builtins to align text, you may find that…
May 30, 2012 by Tom Christiansen
℞ 33: String length in graphemes If you learn nothing else about Unicode, remember this: characters are not bytes are not graphemes are not codepoints. A user-visible symbol (a grapheme) may be composed of multiple codepoints. Multiple combinations of codepoints…
May 29, 2012 by Tom Christiansen
℞ 32: Reverse string by grapheme Because bytes and characters are not isomorphic in Unicode—and what you may see as a user-visible character (a grapheme) is not necessarily a single codepoint in a Unicode string—every string operation must be aware…
May 25, 2012 by Tom Christiansen
℞ 31: Extract by grapheme instead of by codepoint (substr) The Unicode Standard Annex #29 discusses the boundaries between grapheme clusters—what users might perceive as “characters”. The CPAN module Unicode::GCString allows you to treat a Unicode string as a sequence…
May 24, 2012 by Tom Christiansen
℞ 30: Extract by grapheme instead of by codepoint (regex) Remember that Unicode defines a grapheme as “what a user thinks of as a character”. A codepoint is an integer value in the Unicode codespace. While ASCII conflates the two,…
May 22, 2012 by Tom Christiansen
℞ 29: Match Unicode grapheme cluster in regex In the days of ASCII, we spoke of characters and bytes. We saw few differences between them. In the Unicode world, characters are far more than seven bits of data. Far better…
May 21, 2012 by Tom Christiansen
℞ 28: Convert non-ASCII Unicode numerics Unicode digits encompass far more than the ASCII characters 0 - 9. Unless you’ve used /a or /aa, \d matches more than ASCII digits only. That’s good! Unfortunately, Perl’s implicit string-to-number conversion does not…
May 18, 2012 by Tom Christiansen
℞ 27: Unicode normalization Prescription one reminded you to always decompose and recompose Unicode data at the boundaries of your application. Unicode::Normalize can do much more for you. It supports multiple Unicode Normalization Forms. Normalization, of course, takes Unicode data…
May 17, 2012 by Tom Christiansen
℞ 26: Custom character properties Match Unicode Properties in Regex explained that ever Unicode character has one or more properties, specified by the Unicode consortium. You may extend these rule to define your own properties such that Perl can use…
To get in touch, submit an issue to perladvent/perldotcom on GitHub.
This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.
Perl.com and the authors make no representations with respect to the accuracy or completeness of the contents of all work on this website and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. The information published on this website may not be suitable for every situation. All work on this website is provided with the understanding that Perl.com and the authors are not engaged in rendering professional services. Neither Perl.com nor the authors shall be liable for damages arising herefrom.