Support for rendering grapheme clusters #7

sterni commented 2020年08月17日 00:30:33 +02:00

fcft has no support for emoji sequences (at least neither foot nor fuzzel do). This includes:

• Flag sequences (e. g. 🇨🇳)
• Keycap sequences (e. g. 6⃣)
• modifier sequences (e. g. 👋🏻)
• Zero width joiner sequences (e. g. 🧑‍🤝‍🧑)
• Emoji presentation sequences are not rendered obviously wrong, but I suspect the presentation selection is just ignored (haven't tested it though)
• Emoji tag sequences (e. g. 🏴󠁧󠁢󠁥󠁮󠁧󠁿)

I would be interested to hear how feasible it would be to implement these, since I suspect it could be a lot of work or fairly easy if it's simple to get the correct glyph and you only have to check against the three data files relevant.

fcft has no support for emoji sequences (at least neither foot nor fuzzel do). This includes: * Flag sequences (e. g. 🇨🇳) * Keycap sequences (e. g. 6⃣) * modifier sequences (e. g. 👋🏻) * Zero width joiner sequences (e. g. 🧑‍🤝‍🧑) * Emoji presentation sequences are not rendered obviously wrong, but I suspect the presentation selection is just ignored (haven't tested it though) * Emoji tag sequences (e. g. 🏴󠁧󠁢󠁥󠁮󠁧󠁿) I would be interested to hear how feasible it would be to implement these, since I suspect it could be a lot of work or fairly easy if it's simple to get the correct glyph and you only have to check against the three [data files](https://unicode.org/reports/tr51/index.html#Data_Files) relevant.

dnkl commented 2020年08月17日 08:01:26 +02:00

That's correct. fcft currently doesn't parse sequences. Instead, it's the application that does something like:

for character in text-string:
 glyph = fcft_glyph_rasterize(character)

Mostly because this is font shaping, more or less, and so far the idea has been to use e.g. harfbuzz on top of fcft. But neither fuzzel nor foot (or any other programs using fcft) does this. Yet, at least.

Emoji presentation sequences are not rendered obviously wrong, but I suspect the presentation selection is just ignored (haven’t tested it though).

The selection is ignored, yes. This is part of the reason, I think, WHITE FROWNING FACE is rendered in a single cell (and thus cut in half) in foot, instead of two. The other reason being wcwidth() returns 1 for it...

I would be interested to hear how feasible it would be to implement these, since I suspect it could be a lot of work or fairly easy if it’s simple to get the correct glyph and you only have to check against the three data files relevant.

I don't, yet, know the answer to that question. But the way I see it, this has to work in one out of two ways: either e.g. 🇨🇳 has a specific code point, determinable by looking at the Unicode data files. Or, its "code point" is font specific, and you need to use the font's shaping tables to find out what it is.

If the latter, then I'd say the options are to either use harfbuz, or make fcft a full blown shaping enginge.

On the other hand, if it's the first variant, then it would probably be fairly easy to implement some kind of helper function in fcft to do that. Or even introduce a function that takes a whole text string and returns a sequence of glyphs.

Thus, step one: find out how to translate a sequence :)

Slightly related: dnkl/foot#57

That's correct. fcft currently doesn't parse sequences. Instead, it's the application that does something like: ```python for character in text-string: glyph = fcft_glyph_rasterize(character) ``` Mostly because this is font shaping, more or less, and so far the idea has been to use e.g. harfbuzz on top of fcft. But neither fuzzel nor foot (or any other programs using fcft) does this. Yet, at least. > Emoji presentation sequences are not rendered obviously wrong, but I suspect the presentation selection is just ignored (haven’t tested it though). The selection is ignored, yes. This is part of the reason, I think, `WHITE FROWNING FACE` is rendered in a single cell (and thus cut in half) in foot, instead of two. The other reason being `wcwidth()` returns `1` for it... > I would be interested to hear how feasible it would be to implement these, since I suspect it could be a lot of work or fairly easy if it’s simple to get the correct glyph and you only have to check against the three data files relevant. I don't, yet, know the answer to that question. But the way I see it, this has to work in one out of two ways: either e.g. 🇨🇳 has a **specific** code point, determinable by looking at the Unicode data files. Or, its "code point" is font specific, and you need to use the font's shaping tables to find out what it is. If the latter, then I'd say the options are to either use harfbuz, or make fcft a full blown shaping enginge. On the other hand, if it's the first variant, then it would probably be fairly easy to implement some kind of helper function in fcft to do that. Or even introduce a function that takes a whole text string and returns a sequence of glyphs. Thus, step one: find out **how** to translate a sequence :) Slightly related: https://codeberg.org/dnkl/foot/issues/57

sterni commented 2020年08月17日 15:15:44 +02:00

That's correct. fcft currently doesn't parse sequences. Instead, it's the application that does something like:

for character in text-string:
 glyph = fcft_glyph_rasterize(character)

A first step and related issue is probably to do proper Unicode Text Segmentation and deal with grapheme clusters instead of characters (single code points).

I'm curious as to how this works currently, since some grapheme clusters are indeed rendered correctly, but some not. Maybe something else goes wrong for the broken cases?

The attached screenshot (in foot) shows that g̈ is rendered properly, but not நி.

> That's correct. fcft currently doesn't parse sequences. Instead, it's the application that does something like: > > ```python > for character in text-string: > glyph = fcft_glyph_rasterize(character) > ``` A first step and related issue is probably to do proper [Unicode Text Segmentation](https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) and deal with grapheme clusters instead of characters (single code points). I'm curious as to how this works currently, since some grapheme clusters are indeed rendered correctly, but some not. Maybe something else goes wrong for the broken cases? The attached screenshot (in foot) shows that g̈ is rendered properly, but not நி.

dnkl commented 2020年08月17日 19:06:16 +02:00

A first step and related issue is probably to do proper Unicode Text Segmentation and deal with grapheme clusters instead of characters (single code points).

Yup, that needs to be part of the process, definitely.

I’m curious as to how this works currently, since some grapheme clusters are indeed rendered correctly, but some not. Maybe something else goes wrong for the broken cases?

fcft isn't really involved in this:

The attached screenshot (in foot) shows that g̈ is rendered properly

g̈ is a base+combining character. It is actually two glyphs, rendered on top of each other. In e.g. fuzzel, this works out-of-the-box, as long as the font uses negative offsets for the combining characters. Foot deals with it by simply appending zero-width characters to the previous cell: https://codeberg.org/dnkl/foot/src/branch/master/vt.c#L563, but otherwise render it the same way fuzzel does: by rendering multiple glyphs on top of each other (foot does try to recognize wether the font uses negative or positive offsets for the combining characters, so probably has better results than fuzzel for some fonts).

If you look at the code linked above, you'll also see that foot, in this case using fcft, also does precomposition. g̈ cannot be precomposed, but e.g. a+̈ can be precomposed to ä. Pre-composition is currently done by fcft in a helper function - this is something that I would eventually want to remove.

, but not நி.

Here, ி is a spacing combining character, but it isn't zero-width, and hence it isn't appended to the previous cell, but is written to the next cell.

Fuzzel, for example, would need much better support for grapheme clusters to handle e.g. cursor movements correctly, but for glyph lookup, more is needed. But like you said, it is a first step. The question is where this should be implemented. Much of this is perhaps already implemented by e.g. libicu?

It would be a lot less work if we could leverage another library to parse Unicode, and then only deal with mapping grapheme clusters to glyphs in fcft. Anything else feels like feature creep in fcft.

> A first step and related issue is probably to do proper Unicode Text Segmentation and deal with grapheme clusters instead of characters (single code points). Yup, that needs to be part of the process, definitely. > I’m curious as to how this works currently, since some grapheme clusters are indeed rendered correctly, but some not. Maybe something else goes wrong for the broken cases? fcft isn't really involved in this: > The attached screenshot (in foot) shows that g̈ is rendered properly `g̈` is a base+combining character. It is actually two glyphs, rendered on top of each other. In e.g. fuzzel, this works out-of-the-box, as long as the font uses negative offsets for the combining characters. Foot deals with it by simply appending zero-width characters to the previous cell: https://codeberg.org/dnkl/foot/src/branch/master/vt.c#L563, but otherwise render it the same way fuzzel does: by rendering multiple glyphs on top of each other (foot does try to recognize wether the font uses negative or positive offsets for the combining characters, so probably has better results than fuzzel for some fonts). If you look at the code linked above, you'll also see that foot, in this case using fcft, also does precomposition. `g̈` cannot be precomposed, but e.g. `a`+`̈` can be precomposed to `ä`. Pre-composition is currently done by fcft in a helper function - this is something that I would eventually want to remove. > , but not நி. Here, `ி` is a spacing combining character, but it isn't zero-width, and hence it isn't appended to the previous cell, but is written to the next cell. Fuzzel, for example, would need much better support for grapheme clusters to handle e.g. cursor movements correctly, but for glyph lookup, more is needed. But like you said, it is a first step. The question is **where** this should be implemented. Much of this is perhaps already implemented by e.g. libicu? It would be a **lot** less work if we could leverage another library to parse Unicode, and then _only_ deal with mapping grapheme clusters to glyphs in fcft. Anything else feels like feature creep in fcft.

sterni commented 2020年08月17日 19:25:35 +02:00

I agree, implementing grapheme cluster segmentation in fcft is probably a bad idea. libicu has support for segmenting grapheme clusters.

I think this step is pretty clear and simple, the big question then are how is mapping done and do we need to normalize the grapheme clusters for that.

I agree, implementing grapheme cluster segmentation in fcft is probably a bad idea. [libicu has support for segmenting grapheme clusters](http://userguide.icu-project.org/boundaryanalysis). I think this step is pretty clear and simple, the big question then are how is mapping done and do we need to normalize the grapheme clusters for that.

dnkl commented 2020年08月17日 20:26:26 +02:00

I'll also just mention this; foot is going to be much more difficult to this in, since one, it's centered around a grid based layout, and two, it receives data one byte at a time, through a state machine.

(it is also likely to break many applications due to wcwidth() mismatches, which is why I think if/when this support lands in foot it needs to be both compile-time optional - not everyone wants to pull in libicu - and run-time optional via a config option)

Fuzzel is likely a much better test/playground project for this since it is smaller, doesn't use any hacks (precomposition etc) like foot, and only displays "simple" strings.

My, hopefully educated, guess is that mapping is done via font specific tables (if e.g. 🇨🇳 can be mapped to a standardized code point, why aren't we simply using that to begin with?). Which means I would start looking at the ttf/otf tables, how to access them (via FreeType, or directly), and how to use them.

A first step might be to simply load up an emoji font in a font info/creation utility. Perhaps compare it with a different emoji font.

I'll also just mention this; foot is going to be much more difficult to this in, since one, it's centered around a grid based layout, and two, it receives data one byte at a time, through a state machine. (it is also likely to break many applications due to `wcwidth()` mismatches, which is why I think if/when this support lands in foot it needs to be both compile-time optional - not everyone wants to pull in libicu - and run-time optional via a config option) Fuzzel is likely a much better test/playground project for this since it is smaller, doesn't use any hacks (precomposition etc) like foot, and only displays "simple" strings. My, hopefully educated, guess is that mapping is done via font specific tables (if e.g. 🇨🇳 can be mapped to a standardized code point, why aren't we simply using that to begin with?). Which means I would start looking at the ttf/otf tables, how to access them (via FreeType, or directly), and how to use them. A first step might be to simply load up an emoji font in a font info/creation utility. Perhaps compare it with a different emoji font.

sterni commented 2020年08月17日 21:15:41 +02:00

I had a brief (削除) look (削除ここまで) google around.

• For Noto Color Emoji, the font I checked, flags' glyphs are mapped to private use unicode codepoints
• freetype has a specific API for variations sequences

Grapheme clusters don't seem to be handled by freetype, so I tested harfbuzz which from its description seems to be the one: it takes a stream of codepoints and converts them to a stream of (shaped) glyphs.

Testing hb-shape with two fonts, one having the flag glyph and one without it confirms my suspiscion:

[nix-shell:~]$ hb-shape /nix/store/3jycq1i4bf802jpzy5k57af9ns620224-linux-libertine-5.3.0/share/fonts/truetype/public/LinLibertine_R.ttf "🇨🇳"
[.notdef=0+500|.notdef=1+500]
[nix-shell:~]$ hb-shape /nix/store/vdwhbj433rlsdlknwpmyq3sk9gv3x1ac-noto-fonts-emoji-unstable-2019年10月22日/share/fonts/noto/NotoColorEmoji.ttf "🇨🇳"
[gid1456=0+2550]

One thing I'm wondering about, but probably you know it better: harfbuzz is used on the codepoints first, so it seems to be a first step before freetype and is probably hard to integrate with fcft in a calling application?!

At least supporting variation sequences should be possible with only freetype at least.

I had a brief ~~look~~ google around. * For Noto Color Emoji, the font I checked, flags' glyphs are mapped to private use unicode codepoints * freetype has a [specific API for variations sequences](https://www.freetype.org/freetype2/docs/reference/ft2-glyph_variants.html) Grapheme clusters don't seem to be handled by freetype, so I tested harfbuzz which from its description seems to be the one: it takes a stream of codepoints and converts them to a stream of (shaped) glyphs. Testing `hb-shape` with two fonts, one having the flag glyph and one without it confirms my suspiscion: ``` [nix-shell:~]$ hb-shape /nix/store/3jycq1i4bf802jpzy5k57af9ns620224-linux-libertine-5.3.0/share/fonts/truetype/public/LinLibertine_R.ttf "🇨🇳" [.notdef=0+500|.notdef=1+500] [nix-shell:~]$ hb-shape /nix/store/vdwhbj433rlsdlknwpmyq3sk9gv3x1ac-noto-fonts-emoji-unstable-2019年10月22日/share/fonts/noto/NotoColorEmoji.ttf "🇨🇳" [gid1456=0+2550] ``` One thing I'm wondering about, but probably you know it better: harfbuzz is used on the codepoints first, so it seems to be a first step *before* freetype and is probably hard to integrate with fcft in a calling application?! At least supporting variation sequences should be possible with only freetype at least.

dnkl commented 2020年08月18日 06:39:12 +02:00

Grapheme clusters don’t seem to be handled by freetype, so I tested harfbuzz which from its description seems to be the one: it takes a stream of codepoints and converts them to a stream of (shaped) glyphs.

Yup, that's why I mentioned harfbuzz earlier; it's the goto for text shaping :)

One thing I’m wondering about, but probably you know it better: harfbuzz is used on the codepoints first, so it seems to be a first step before freetype and is probably hard to integrate with fcft in a calling application?!

Up until now, my thoughts have been on line "applications use harfbuzz, then call fcft". But after this discussion I think that wouldn't work. It (harfbuzz) has to be called from fcft, because fcft actually manages many fonts for each fcft font instance: the primary font, and all it's fallback fonts. So when mapping a grapheme cluster to a code point, it may have to try harfbuzz on several fonts before finding one. This isn't something an application can do.

This is probably way over-simplified, but right now I'm thinking of an API along these lines:

We already have fcft_glyph_rasterize(), which takes one character, and returns one glyph.

I'm hoping we can add something like:

glyph_list_t fcft_glyph_rasterize_grapheme_cluster(const wchar_t *cluster, size_t len)
{
 for font in [primary + fallbacks] {
 if (font.has_grapheme_cluster(cluster))
 return fcft_glyph_rasterize(harfbuzz_shape(cluster));
 }
}

In theory, the API could take any string, not just a grapheme cluster. But, I'd like add caching of the grapheme cluster mapping, which means such an API would have to be layered on top of fcft_glyph_rasterize_grapheme_cluster().

> Grapheme clusters don’t seem to be handled by freetype, so I tested harfbuzz which from its description seems to be the one: it takes a stream of codepoints and converts them to a stream of (shaped) glyphs. Yup, that's why I mentioned harfbuzz earlier; it's _the_ goto for text shaping :) > One thing I’m wondering about, but probably you know it better: harfbuzz is used on the codepoints first, so it seems to be a first step before freetype and is probably hard to integrate with fcft in a calling application?! Up until now, my thoughts have been on line "applications use harfbuzz, then call fcft". But after this discussion I think that wouldn't work. It (harfbuzz) _has_ to be called from fcft, because fcft actually manages many fonts for each fcft font instance: the primary font, and all it's fallback fonts. So when mapping a grapheme cluster to a code point, it may have to try harfbuzz on several fonts before finding one. This isn't something an application can do. This is probably way over-simplified, but right now I'm thinking of an API along these lines: We already have `fcft_glyph_rasterize()`, which takes **one** _character_, and returns **one** _glyph_. I'm hoping we can add something like: ```c glyph_list_t fcft_glyph_rasterize_grapheme_cluster(const wchar_t *cluster, size_t len) { for font in [primary + fallbacks] { if (font.has_grapheme_cluster(cluster)) return fcft_glyph_rasterize(harfbuzz_shape(cluster)); } } ``` In theory, the API _could_ take any **string**, not just a grapheme cluster. But, I'd like add caching of the grapheme cluster mapping, which means such an API would have to be layered on top of `fcft_glyph_rasterize_grapheme_cluster()`.

dnkl commented 2020年08月18日 18:01:05 +02:00

I've glanced over the harfbuzz APIs, and it seems very doable. Up next is implementing a POC, I think.

As for grapheme cluster segmentation, ICU does the job, but it's a bit clunky. Among other things, its break iterator API only works with UTF-16 encoded strings, something neither fuzzel nor foot uses internally. Meaning we have to first convert strings to UTF-16, and then map the cluster break indices back to wchar indices.

This is particulary bad for foot since it will be really slow. It also would have been much better if it had been possible to push one code point at the time to the segmentation functions, since that's how foot receives data. I haven't been able to find anything better, so perhaps we'll have to write it ourselves.

Anyway, that doesn't really have anything to do with the fcft changes, but is just a heads up.

I've glanced over the harfbuzz APIs, and it seems very doable. Up next is implementing a POC, I think. As for grapheme cluster segmentation, ICU _does_ the job, but it's a bit clunky. Among other things, its break iterator API **only** works with UTF-16 encoded strings, something neither fuzzel nor foot uses internally. Meaning we have to first convert strings to UTF-16, and then map the cluster break indices back to wchar indices. This is particulary bad for foot since it will be really slow. It also would have been much better if it had been possible to push one code point at the time to the segmentation functions, since that's how foot receives data. I haven't been able to find anything better, so perhaps we'll have to write it ourselves. Anyway, that doesn't really have anything to do with the fcft changes, but is just a heads up.

sterni commented 2020年08月18日 18:07:04 +02:00

I’ve glanced over the harfbuzz APIs, and it seems very doable.

Yeah, the API also looked very nice to me.

This is particulary bad for foot since it will be really slow. It also would have been much better if it had been possible to push one code point at the time to the segmentation functions, since that’s how foot receives data.

That's a real shame, but I think it's solvable. It is definitely possible, probably just a matter of finding a library for the job that isn't unmaintained. I tried to search for something in the vain of Rust's unicode-segmentation crate, but unsuccessfully so far.

Up next is implementing a POC, I think.

I'd really like to help out, but at the moment I should really start focussing on my paper for university, but I'd be happy to give feedback etc. :)

> I’ve glanced over the harfbuzz APIs, and it seems very doable. Yeah, the API also looked very nice to me. > This is particulary bad for foot since it will be really slow. It also would have been much better if it had been possible to push one code point at the time to the segmentation functions, since that’s how foot receives data. That's a real shame, but I think it's solvable. It is definitely possible, probably just a matter of finding a library for the job that isn't unmaintained. I tried to search for something in the vain of Rust's [unicode-segmentation](https://github.com/unicode-rs/unicode-segmentation) crate, but unsuccessfully so far. > Up next is implementing a POC, I think. I'd really like to help out, but at the moment I should *really* start focussing on my paper for university, but I'd be happy to give feedback etc. :)

dnkl commented 2020年08月18日 18:11:44 +02:00

I’d really like to help out, but at the moment I should really start focussing on my paper for university, but I’d be happy to give feedback etc. :)

Just having someone to discuss with, and who reads the code helps a lot :)

> I’d really like to help out, but at the moment I should really start focussing on my paper for university, but I’d be happy to give feedback etc. :) Just having someone to discuss with, and who reads the code helps a lot :)

dnkl commented 2020年09月15日 20:04:04 +02:00

Grapheme shaping support has now been merged to master. We still do not support shaping whole text runs, which means e.g. no ligatures.

This is tracked in #12.

Grapheme shaping support has now been merged to master. We still do not support shaping whole text runs, which means e.g. no ligatures. This is tracked in https://codeberg.org/dnkl/fcft/issues/12.