Bidirectional text support #756

CIAvash commented 2021年10月20日 14:56:56 +02:00

• Foot Version:
  foot version: 1.9.2-39-g6dfacb9c (Oct 20 2021, branch 'master') +pgo +ime +graphemes
• Log output:

  info: main.c:402: version: 1.9.2-39-g6dfacb9c (Oct 20 2021, branch 'master') +pgo +ime +graphemes
  info: main.c:409: arch: Linux x86_64/64-bit
  info: main.c:416: locale: en_US.UTF-8
  info: config.c:3064: loading configuration from /home/siavash/.config/foot/foot.ini
  info: fcft.c:267: fcft: 2.4.5 +graphemes +runs
  info: fcft.c:277: fontconfig: 2.13.94
  info: fcft.c:283: freetype: 2.11.0
  info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Regular.ttf: size=8.00pt/8px, dpi=75.00
  info: wayland.c:1275: DVI-I-1: 1920x1200+1920x0@60Hz SyncMaster 21.95" scale=1 PPI=106x109 (physical) PPI=106x109 (logical), DPI=103.14
  info: wayland.c:1275: HDMI-A-1: 1920x1080+0x0@60Hz ASUS VS228 21.68" scale=1 PPI=106x108 (physical) PPI=106x108 (logical), DPI=101.60
  info: wayland.c:1428: requesting SSD decorations
  info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Regular.ttf: size=10.00pt/14px, dpi=103.14
  info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Italic.ttf: size=10.00pt/14px, dpi=103.14
  info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-BoldItalic.ttf: size=10.00pt/14px, dpi=103.14
  info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Bold.ttf: size=10.00pt/14px, dpi=103.14
  info: terminal.c:694: cell width=8, height=19
  info: terminal.c:583: using 2 rendering threads
  info: wayland.c:805: using SSD decorations
  info: fcft.c:758: /usr/share/fonts/OTF/PowerlineSymbols.otf: size=10.00pt/14px, dpi=103.14
  info: wayland.c:1586: cursor theme: (null), size: 24, scale: 1
  info: fcft.c:758: /usr/share/fonts/TTF/DejaVuSansMono.ttf: size=10.00pt/14px, dpi=103.14
  

• Wayland compositor: Sway 1.6.1
• Steps to reproduce:

  echo 'متن راست به چپ'
  

I saw that foot uses fcft as a dependency which uses Harfbuzz for text shaping, so I thought text shaping for right to left languages would work, but I tried and it didn't.
Letters do not get combined, and direction of the text is RTL instead of LTR.

From the image in fcft's README, it seems fcft_text_run_rasterize() should at least shape it correctly(if not set its direction correctly). I did a quick search, it seems it's being used in render_osd.

RTL text(Persian(Arabic script)) in VTE based terminals:
RTL text in VTE based terminal
RTL text in foot:
RTL text in foot

- Foot Version: foot version: 1.9.2-39-g6dfacb9c (Oct 20 2021, branch 'master') +pgo +ime +graphemes - Log output: ``` info: main.c:402: version: 1.9.2-39-g6dfacb9c (Oct 20 2021, branch 'master') +pgo +ime +graphemes info: main.c:409: arch: Linux x86_64/64-bit info: main.c:416: locale: en_US.UTF-8 info: config.c:3064: loading configuration from /home/siavash/.config/foot/foot.ini info: fcft.c:267: fcft: 2.4.5 +graphemes +runs info: fcft.c:277: fontconfig: 2.13.94 info: fcft.c:283: freetype: 2.11.0 info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Regular.ttf: size=8.00pt/8px, dpi=75.00 info: wayland.c:1275: DVI-I-1: 1920x1200+1920x0@60Hz SyncMaster 21.95" scale=1 PPI=106x109 (physical) PPI=106x109 (logical), DPI=103.14 info: wayland.c:1275: HDMI-A-1: 1920x1080+0x0@60Hz ASUS VS228 21.68" scale=1 PPI=106x108 (physical) PPI=106x108 (logical), DPI=101.60 info: wayland.c:1428: requesting SSD decorations info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Regular.ttf: size=10.00pt/14px, dpi=103.14 info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Italic.ttf: size=10.00pt/14px, dpi=103.14 info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-BoldItalic.ttf: size=10.00pt/14px, dpi=103.14 info: fcft.c:758: /usr/share/fonts/TTF/RobotoMono-Bold.ttf: size=10.00pt/14px, dpi=103.14 info: terminal.c:694: cell width=8, height=19 info: terminal.c:583: using 2 rendering threads info: wayland.c:805: using SSD decorations info: fcft.c:758: /usr/share/fonts/OTF/PowerlineSymbols.otf: size=10.00pt/14px, dpi=103.14 info: wayland.c:1586: cursor theme: (null), size: 24, scale: 1 info: fcft.c:758: /usr/share/fonts/TTF/DejaVuSansMono.ttf: size=10.00pt/14px, dpi=103.14 ``` - Wayland compositor: Sway 1.6.1 - Steps to reproduce: ```console echo 'متن راست به چپ' ``` I saw that foot uses fcft as a dependency which uses Harfbuzz for text shaping, so I thought text shaping for right to left languages would work, but I tried and it didn't. Letters do not get combined, and direction of the text is RTL instead of LTR. From the image in fcft's README, it seems `fcft_text_run_rasterize()` should at least shape it correctly(if not set its direction correctly). I did a quick search, it seems it's being used in [`render_osd`](https://codeberg.org/dnkl/foot/src/branch/master/render.c#L1604). RTL text(Persian(Arabic script)) in VTE based terminals:  RTL text in foot: 

dnkl commented 2021年10月20日 19:13:50 +02:00

Related to #57

Unfortunately, there's much more to this than just calling the right fcft function. Foot currently has zero support for text shaping across cells. This includes ligatures and RTL scripts.

I guess that, since the ligatures issue is still open, we can keep this one open too. But be warned: there's no plan to implement this. I'm not saying it will never happen, or that it is impossible. But it is a difficult problem to solve.

Related to https://codeberg.org/dnkl/foot/issues/57 Unfortunately, there's much more to this than just calling the right fcft function. Foot currently has zero support for text shaping across cells. This includes ligatures and RTL scripts. I guess that, since the ligatures issue is still open, we can keep this one open too. But be warned: **there's no plan to implement this**. I'm not saying it will never happen, or that it is impossible. But it is a difficult problem to solve.

tinywrkb commented 2022年01月24日 14:34:42 +01:00

BiDi in the terminal is much more complex than throwing UAX9 (basically harfbuzz) on it.
Some work have been done here to create a standard, but there are a bunch of unresolved issues.

IMO doing BiDi in the regular terminal mode is wrong, and should be avoided unless an application explicitly notify the terminal emulator that the text has formatting characters, and directional embedding characters were added.

Applications that don't use the regular mode (alternate screen/fullscreen apps), like text editors and file managers, are layout-aware, and they can implement BiDI support internally, though line-breaking still sounds to me like a unresolved issue, especially considering Arabic script shaping.

To the unaware LTR script user, experienced RTL script users can do fine by reading reversed text, characters in logical order without BiDI algorithm being applied.
You get this superpower after staring enough time at the terminal, and it's more practical than having to figure out what the heck the terminal emulator did when it applied the BiDi algorithm.

BiDi in the terminal is much more complex than throwing UAX9 (basically harfbuzz) on it. Some work have been done [here](https://terminal-wg.pages.freedesktop.org/bidi/) to create a standard, but there are a [bunch of unresolved issues](https://gitlab.freedesktop.org/terminal-wg/bidi/-/issues). IMO doing BiDi in the regular terminal mode is wrong, and should be avoided unless an application explicitly notify the terminal emulator that the text has formatting characters, and directional embedding characters were added. Applications that don't use the regular mode (alternate screen/fullscreen apps), like text editors and file managers, are layout-aware, and they can implement BiDI support internally, though line-breaking still sounds to me like a unresolved issue, especially considering Arabic script shaping. To the unaware LTR script user, experienced RTL script users can do fine by reading reversed text, characters in logical order without BiDI algorithm being applied. You get this superpower after staring enough time at the terminal, and it's more practical than having to figure out what the heck the terminal emulator did when it applied the BiDi algorithm.

CIAvash commented 2022年01月24日 18:50:55 +01:00

For output, one can pipe the stdout to fribidi command line tool to make RTL text show correctly, instead of staring at the terminal.

For output, one can pipe the `stdout` to `fribidi` command line tool to make RTL text show correctly, instead of staring at the terminal.

tinywrkb commented 2022年01月24日 19:15:01 +01:00

For output, one can pipe the stdout to fribidi command line tool to make RTL text show correctly, instead of staring at the terminal.

No, that assumption is wrong! You can't magically make a mix of RTL and LTR text displayed correctly by throwing it at fribidi.

I'm not touching paragraph/line alignment, it's not really matter for BiDi.
Capital is strong RTL, lower is strong LTR.
Line direction must be LTR, because if it would be RTL, then logical ls -l ABC abc becomes display abc CBA ls -l.

Let's say you have an output with a path, let's say it's rsync log.
Logical is /ABC/DEF/ghi. so display is /FED/CBAghi.
How exactly do you expect the user to know what files have been changed from this nonsense mess?

> For output, one can pipe the stdout to fribidi command line tool to make RTL text show correctly, instead of staring at the terminal. No, that assumption is wrong! You can't magically make a mix of RTL and LTR text displayed correctly by throwing it at fribidi. I'm not touching paragraph/line alignment, it's not really matter for BiDi. Capital is strong RTL, lower is strong LTR. Line direction must be LTR, because if it would be RTL, then logical `ls -l ABC abc` becomes display `abc CBA ls -l`. Let's say you have an output with a path, let's say it's rsync log. Logical is `/ABC/DEF/ghi`. so display is `/FED/CBAghi`. How exactly do you expect the user to know what files have been changed from this nonsense mess?

CIAvash commented 2022年01月24日 20:18:47 +01:00

fribidi seems to do fine for me on your examples.

By default, fribidi uses --wltr(Set base direction to LTR if no strong character found)

So the problem occures when LTR text is prepended to RTL text(/abc/DEF/GHI). But in this case, other applications seem to do the same thing.

Besided, I didn't mention fribidi as a perfect solution, It's just better than nothing, and is perfect for some cases.

`fribidi` seems to do fine for me on your examples. By default, `fribidi` uses `--wltr`(Set base direction to LTR if no strong character found) So the problem occures when LTR text is prepended to RTL text(/abc/DEF/GHI). But in this case, other applications seem to do the same thing. Besided, I didn't mention `fribidi` as a perfect solution, It's just better than nothing, and is perfect for some cases.

CIAvash commented 2022年03月20日 08:01:47 +01:00

For reference: https://github.com/wez/wezterm/issues/784

For reference: https://github.com/wez/wezterm/issues/784