Composite emoji (flags and some others) not shown correctly #516

y0ast commented 2021年05月14日 12:40:35 +02:00

Thanks for foot, it's a great terminal!

There's a problem with showing emoji flags. For example:

🇿🇲 flag: Zambia
🇿🇼 flag: Zimbabwe
🏴󠁧󠁢󠁥󠁮󠁧󠁿 flag: England

Are not rendered correctly (tested by cat'ing a file with those lines). The black/white flag work fine, but all country flags are broken (showing letter emoji), and the UK countries show a black flag.

It seems that flags are created by combining two regional character emojis. For example, vietnam is: U+1F1FB U+1F1F3 or 🇻 🇳 (removing the space between them turns it into the vietnam flag 🇻🇳 in firefox)

The smiling face emoji has a related problem:
☺️ (U+263A, U+FE0F) is rendered as ☺ (U+263A) - the variant selector U+FE0F is ignored by foot.

Kitty gets this right (although not the uk flags), but alacritty has the same problem. iterm2/terminal on MacOS also do this right (including uk flags). Rendering in monospace in firefox is also fine.

Thanks for `foot`, it's a great terminal! There's a problem with showing emoji flags. For example: ``` 🇿🇲 flag: Zambia 🇿🇼 flag: Zimbabwe 🏴󠁧󠁢󠁥󠁮󠁧󠁿 flag: England ``` Are not rendered correctly (tested by `cat`'ing a file with those lines). The black/white flag work fine, but all country flags are broken (showing letter emoji), and the UK countries show a black flag. It seems that flags are created by combining two regional character emojis. For example, vietnam is: U+1F1FB U+1F1F3 or 🇻 🇳 (removing the space between them turns it into the vietnam flag 🇻🇳 in firefox) The smiling face emoji has a related problem: ☺️ (U+263A, U+FE0F) is rendered as ☺ (U+263A) - the variant selector U+FE0F is ignored by foot. Kitty gets this right (although not the uk flags), but alacritty has the same problem. iterm2/terminal on MacOS also do this right (including uk flags). Rendering in monospace in firefox is also fine.

dnkl commented 2021年05月14日 13:22:53 +02:00

This is unfortunately a known limitation, as it requires text shaping support.

The smiling face emoji has a related problem:

FWIW, this depends on which font the glyph gets loaded from. I.e. it is possible to configure foot to load these from the "correct" (emoji) font. But yeah, the real solution is to stop ignoring the variant selector.

Kitty gets this right (although not the uk flags), but alacritty has the same problem. iterm2/terminal on MacOS also do this right (including uk flags).

In terminals, this is bigger than "just" text shaping; the terminal and the applications running in it need to agree on how many cells a grapheme occupies. The easiest way to ensure that is to use the same interface - method - to measure that. And that typically comes down to the wcwidth() an wcswidth() functions. And these don't handle e.g. emoji sequences.

What kitty and the others that "support" this are doing, is they provide their own implementation of wcswidth(). As a result, you can easily "break" e.g. the prompt (cursor gets "out of sync"), by using these emoji sequences.

Duplicate of: #57

You might also be interrested in reading through #100

This is unfortunately a known limitation, as it requires text shaping support. > The smiling face emoji has a related problem: FWIW, this depends on which font the glyph gets loaded from. I.e. it is possible to configure foot to load these from the "correct" (emoji) font. But yeah, the _real_ solution is to stop ignoring the variant selector. > Kitty gets this right (although not the uk flags), but alacritty has the same problem. iterm2/terminal on MacOS also do this right (including uk flags). In terminals, this is bigger than "just" text shaping; the terminal and the applications running in it need to agree on how many cells a grapheme occupies. The easiest way to ensure that is to use the same interface - method - to measure that. And that typically comes down to the `wcwidth()` an `wcswidth()` functions. And these don't handle e.g. emoji sequences. What kitty and the others that "support" this are doing, is they provide their own implementation of `wcswidth()`. As a result, you can easily "break" e.g. the prompt (cursor gets "out of sync"), by using these emoji sequences. Duplicate of: https://codeberg.org/dnkl/foot/issues/57 You might also be interrested in reading through https://codeberg.org/dnkl/foot/pulls/100

y0ast commented 2021年05月14日 13:47:54 +02:00

Thanks for your reply, it seems like #100 would solve this issue.

I also found:
dnkl/fcft#7

FWIW, this depends on which font the glyph gets loaded from. I.e. it is possible to configure foot to load these from the "correct" (emoji) font. But yeah, the real solution is to stop ignoring the variant selector.

The variant selector is significantly more elegant that the blacklisting of unicode characters. Maybe worth adding to the wiki that a "true" solution does exist, but isn't implemented (yet).

http://unicode.org/emoji/charts/emoji-variants.html

What kitty and the others that "support" this are doing, is they provide their own implementation of wcswidth(). As a result, you can easily "break" e.g. the prompt (cursor gets "out of sync"), by using these emoji sequences.

I see. Well, at the moment if I paste ☺️ into foot, the cursor also goes bananas. Although I suspect it's a fish issue, as bash is fine. This doesn't happen with kitty+fish, but does happen with alacritty again!

Thanks for your reply, it seems like #100 would solve this issue. I also found: https://codeberg.org/dnkl/fcft/issues/7 > FWIW, this depends on which font the glyph gets loaded from. I.e. it is possible to configure foot to load these from the "correct" (emoji) font. But yeah, the real solution is to stop ignoring the variant selector. The variant selector is significantly more elegant that the blacklisting of unicode characters. Maybe worth adding to the wiki that a "true" solution does exist, but isn't implemented (yet). http://unicode.org/emoji/charts/emoji-variants.html > What kitty and the others that "support" this are doing, is they provide their own implementation of wcswidth(). As a result, you can easily "break" e.g. the prompt (cursor gets "out of sync"), by using these emoji sequences. I see. Well, at the moment if I paste ☺️ into foot, the cursor also goes bananas. Although I suspect it's a `fish` issue, as `bash` is fine. This doesn't happen with kitty+fish, but does happen with alacritty again!

dnkl commented 2021年05月14日 13:54:31 +02:00

This doesn't happen with kitty+fish, but does happen with alacritty again!

Perhaps fish has been updated to use a "more" correct wcwidth(), just like kitty. Must be relatively new:

"For example kitty with its heuristics has this problem: Everytime I paste a emoji into a command my fish prompt gets messed up completely to the point I can't really edit it without guesswork anymore."

So yeah, no matter what we do, we break something. This is why I'd really like to see this being solved in standard interfaces, rather than every application (and terminal!) doing its own thing :/

I also found: dnkl/fcft#7

fcft-2.4 actually has support for shaping entire text runs, and this will be used in my other projects. When that has been implemented, they will be able to render all of these sequences correctly.

foot is different beast though, with the wcwidth() issue being the biggest, but not the only, problem.

> This doesn't happen with kitty+fish, but does happen with alacritty again! Perhaps fish has been updated to use a "more" correct `wcwidth()`, just like kitty. Must be relatively new: > _"For example kitty with its heuristics has this problem: Everytime I paste a emoji into a command my fish prompt gets messed up completely to the point I can't really edit it without guesswork anymore."_ So yeah, no matter what we do, we break something. This is why I'd really like to see this being solved in standard interfaces, rather than every application (and terminal!) doing its own thing :/ > I also found: dnkl/fcft#7 fcft-2.4 actually has support for shaping entire text runs, and this will be used in my other projects. When that has been implemented, they will be able to render all of these sequences correctly. foot is different beast though, with the `wcwidth()` issue being the biggest, but not the only, problem.

y0ast commented 2021年05月14日 14:07:00 +02:00

It seems fish has made their implementation open domain: https://github.com/ridiculousfish/widecharwidth

It seems that fish autocomplete computes the wrong length for the autocompletion string (using length 2 for ☺️ instead of the 1 width ☺ that foot renders) , which has the effect of a backspace (that goes into the previous line....)

It seems fish has made their implementation open domain: https://github.com/ridiculousfish/widecharwidth It seems that `fish` autocomplete computes the wrong length for the autocompletion string (using length 2 for ☺️ instead of the 1 width ☺ that foot renders) , which has the effect of a backspace (that goes into the previous line....)

dnkl commented 2021年05月14日 14:18:55 +02:00

It seems that fish autocomplete computes the wrong length for the autocompletion string (using length 2 for ☺️ instead of the 1 width ☺ that foot renders) , which has the effect of a backspace (that goes into the previous line....)

For a fully qualified SMILING FACE, 2 is probably the correct width. For an unqualified I believe 1 would be more correct, in this particular case.

In any case, this is exactly the problem; fish and foot don't agree on the width, and that leads to issues. If foot is changed to match fish, bash breaks instead 🤷.

> It seems that fish autocomplete computes the wrong length for the autocompletion string (using length 2 for ☺️ instead of the 1 width ☺ that foot renders) , which has the effect of a backspace (that goes into the previous line....) For a fully qualified SMILING FACE, 2 is probably the correct width. For an unqualified I believe 1 would be more correct, in this particular case. In any case, this is exactly the problem; fish and foot don't agree on the width, and that leads to issues. If foot is changed to match fish, bash breaks instead 🤷.