XTGETTCAP doesn't encode control-characters (i.e. ^H) correctly #1701

mitchellh commented 2024年04月21日 05:45:47 +02:00

Hello! 👋 This is based on an issue that was reported to me over in Ghostty.

XTGETTCAP for ht ("^I"):

• xterm: 09
• foot: 5E49
• kitty: 5E49

Note, I had to patch xterm to support ht but you can use kbs if you want a built-in xterm terminfo value that uses this format. But kbs for xterm is ^H and for foot its ^? so don't expect them to match, but you can see the encoding happening.[1]

The logic in Foot seems to be that for parameterized strings, they are returned in terminfo source format (matching Kitty). But for non-parameterized strings, they are returned in termcap encoded format (i.e. \E is replaced with 0x1B). termcap encoded format should also process control characters such as ^H and xterm behavior seems to match this, but foot does not.

Note that termcap encoding is way more complicated and at least for my terminal I've opted to say "fuck that" and only handle the encodings that I actually use. So, there are still encodings that we're all probably still wrong on but they don't matter in practice because our terminfo doesn't have them.

Ghostty used to also do 5E49 but I've now fixed this to match xterm to process control characters to 09.

As a disclaimer, I may be missing some detail about XTGETTCAP that makes this behavior wrong, but in my studying of ncurses, xterm, and foot+kitty this is the conclusion I came to. Please correct me if I'm wrong.

[1]: Separate to this issue, why is foot's terminfo ^? for kbs? Shouldn't this be ^H?

Hello! 👋 This is based on an issue that was reported to me over in Ghostty. `XTGETTCAP` for `ht` ("^I"): * xterm: `09` * foot: `5E49` * kitty: `5E49` Note, I had to patch xterm to support `ht` but you can use `kbs` if you want a built-in xterm terminfo value that uses this format. But `kbs` for xterm is `^H` and for foot its `^?` so don't expect them to match, but you can see the encoding happening.[1] The logic in Foot seems to be that for parameterized strings, they are returned in terminfo source format (matching Kitty). But for non-parameterized strings, they are returned in termcap encoded format (i.e. `\E` is replaced with 0x1B). termcap encoded format should also process control characters such as `^H` and xterm behavior seems to match this, but foot does not. Note that termcap encoding is [way more complicated](https://github.com/mirror/ncurses/blob/master/ncurses/tinfo/captoinfo.c#L619) and at least for my terminal I've opted to say "fuck that" and only handle the encodings that I actually use. So, there are still encodings that we're all probably still wrong on but they don't matter in practice because our terminfo doesn't have them. Ghostty used to also do `5E49` but I've now fixed this to match xterm to process control characters to `09`. As a disclaimer, I may be missing some detail about XTGETTCAP that makes this behavior wrong, but in my studying of ncurses, xterm, and foot+kitty this is the conclusion I came to. Please correct me if I'm wrong. [1]: Separate to this issue, why is foot's terminfo `^?` for `kbs`? Shouldn't this be `^H`?

craigbarnes commented 2024年04月21日 13:02:40 +02:00

Separate to this issue, why is foot's terminfo ^? for kbs? Shouldn't this be ^H?

https://codeberg.org/dnkl/foot#backspace

> Separate to this issue, why is foot's terminfo ^? for kbs? Shouldn't this be ^H? https://codeberg.org/dnkl/foot#backspace

mitchellh commented 2024年04月21日 18:07:33 +02:00

Ah right, sorry, it was late and I was getting all spinned up reading all these terminfo sources lol. That makes sense.

Ah right, sorry, it was late and I was getting all spinned up reading all these terminfo sources lol. That makes sense.

craigbarnes commented 2024年04月22日 15:05:51 +02:00

In my view, returning string capabilities in terminfo source format seems wrong for multiple reasons:

1. It's not in alignment with xterm's basic description of XTGETTCAP:

   xterm responds with [...] and the value of the corresponding string that xterm would send

2. It places the burden of decoding (some of) the source format on the application (and ESC has 4+ possible encodings, if I'm not mistaken)
3. It's not the same string as would be returned by the "traditional" libtinfo API (i.e. tigetstr(3)) - if applications can (and already do) handle those strings, I don't see any reason for them to be sent source format strings in XTGETTCAP replies
4. The hex encoding was clearly imposed because ESC can't appear in DCS strings, so replying with hex-encoded source format strings seems like it just constitutes double-escaping, for no apparent reason

Given the state of xterm's limited implementation and docs, I think the closest thing to a "source of truth" would be the tigetstr(3) API (or tput(1), as a convenient proxy for it):

$ tput -Tfoot kbs | hexdump -C
00000000 7f |.|
00000001
$ tput -Tfoot kf1 | hexdump -C
00000000 1b 4f 50 |.OP|
00000003
$ tput -Tfoot Ms | hexdump -C
00000000 1b 5d 35 32 3b 25 70 31 25 73 3b 25 70 32 25 73 |.]52;%p1%s;%p2%s|
00000010 1b 5c |.\|
00000012
$ tput -Tfoot ht | hexdump -C
00000000 09 |.|
00000001

In my view, returning string capabilities in terminfo source format seems wrong for multiple reasons: 1. It's not in alignment with xterm's basic [description](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h4-Device-Control-functions:DCS-plus-q-Pt-ST.F95) of `XTGETTCAP`: > xterm responds with [...] and the value of the corresponding **string that xterm would send** 2. It places the burden of decoding (some of) the source format on the application (and `ESC` has 4+ possible encodings, if I'm not mistaken) 3. It's not the same string as would be returned by the "traditional" libtinfo API (i.e. [`tigetstr(3)`]) - if applications can (and already do) handle those strings, I don't see any reason for them to be sent source format strings in `XTGETTCAP` replies 4. The hex encoding was clearly imposed because `ESC` can't appear in DCS strings, so replying with hex-encoded source format strings seems like it just constitutes double-escaping, for no apparent reason Given the state of xterm's limited implementation and docs, I think the closest thing to a "source of truth" would be the [`tigetstr(3)`] API (or [`tput(1)`], as a convenient proxy for it): ```sh $ tput -Tfoot kbs | hexdump -C 00000000 7f |.| 00000001 $ tput -Tfoot kf1 | hexdump -C 00000000 1b 4f 50 |.OP| 00000003 $ tput -Tfoot Ms | hexdump -C 00000000 1b 5d 35 32 3b 25 70 31 25 73 3b 25 70 32 25 73 |.]52;%p1%s;%p2%s| 00000010 1b 5c |.\| 00000012 $ tput -Tfoot ht | hexdump -C 00000000 09 |.| 00000001 ``` [`tigetstr(3)`]: https://man7.org/linux/man-pages/man3/curs_terminfo.3x.html [`tput(1)`]: https://www.man7.org/linux/man-pages/man1/tput.1.html

craigbarnes commented 2024年04月22日 15:34:48 +02:00

@mitchellh I'm in broad agreement with what you said above and in particular with the idea that emitting control characters in their (hex-encoded) compiled form seems like the right thing to do.

@mitchellh I'm in broad agreement with what you said above and in particular with the idea that emitting control characters in their (hex-encoded) compiled form seems like the right thing to do.

dnkl commented 2024年04月23日 08:07:27 +02:00

I do think always returning tigetstr encoded values would be much cleaner.

Just need to figure out the best way to implement that...

I do think always returning `tigetstr` encoded values would be much cleaner. Just need to figure out the best way to implement that...

dnkl commented 2024年04月23日 08:25:08 +02:00

Ok, so looking at our implementation, it looks like all we need to do is

• Stop special casing parametrized capabilities:
  scripts/generate-builtin-terminfo.py

  Lines 51 to 75 in 4d4ef5e

  class StringCapability(Capability):

  def __init__(self, name: str, value: str):

  # Expand \E to literal ESC in non-parameterized capabilities

  if '%' not in value:

  # Ensure e.g. \E7 doesn’t get translated to "0337円", which

  # would be interpreted as octal 337 by the C compiler

  value = re.sub(r'\\E([0-7])', r'\\033""\1', value)

  # Replace \E with an actual escape

  value = re.sub(r'\\E', r'\\033', value)

  # Don’t escape ‘:’

  value = value.replace('\\:', ':')

  else:

  value = value.replace("\\", "\\\\")

  # # Need to double-escape backslashes. These only occur in

  # # ‘\E\’ combos. Note that \E itself is updated below

  # value = value.replace('\\E\\\\', '\\E\\\\\\\\')

  # # Need to double-escape \E in C string literals

  # value = value.replace('\\E', '\\\\E')

  super().__init__(name, value)
• Recognize the source variant of control characters, and translate them to actual control characters.

Ok, so looking at our implementation, it looks like all we need to do is * Stop special casing parametrized capabilities: https://codeberg.org/dnkl/foot/src/commit/4d4ef5eed536c373f93c766f4071e14665a9b00e/scripts/generate-builtin-terminfo.py#L51-L75 * Recognize the source variant of control characters, and translate them to actual control characters.

dnkl commented 2024年04月23日 08:26:26 +02:00

Though, not sure if this covers all capabilities...

Though, not sure if this covers _all_ capabilities...

dnkl commented 2024年04月23日 08:49:34 +02:00

I've compared a couple of capabilities, including parametrized and un-parametrized capabilities, capabilities with control characters, and as far as I can tell, we'll always match tigetstr if we do the changes mentioned above.

I've compared a couple of capabilities, including parametrized and un-parametrized capabilities, capabilities with control characters, and as far as I can tell, we'll always match `tigetstr` if we do the changes mentioned above.

craigbarnes commented 2024年04月23日 15:11:35 +02:00

Though, not sure if this covers all capabilities...

I think the "Types of Capabilities" section in the terminfo(5) man pages lists all the constructs that generate-builtin-terminfo.py might need to handle. In summary, basically the following:

• Special character escapes (e.g. \e, \E, \n, \:, full list here)
• Caret notation (in the range ^@ through ^_ and ^?)
• Octal escapes (e.g. 033円)

...or in practice just these:

$ LANG=C grep -Eo '(\\[:,^0\\beEflnrst]|\^[@-_?]|\\[0-7]+)' foot.info | sort | uniq -c | sort -nr
 266 \E
 25 \:
 16 \\
 2 \n
 1 \r
 1 ^I
 1 ^H
 1 ^G
 1 ^?

> Though, not sure if this covers all capabilities... I think the "Types of Capabilities" section in the [`terminfo(5)`] man pages lists all the constructs that `generate-builtin-terminfo.py` might need to handle. In summary, basically the following: * Special character escapes (e.g. `\e`, `\E`, `\n`, `\:`, [full list here][unescape_char]) * Caret notation (in the range `^@` through `^_` and `^?`) * Octal escapes (e.g. `033円`) ...or in practice just these: ```sh $ LANG=C grep -Eo '(\\[:,^0\\beEflnrst]|\^[@-_?]|\\[0-7]+)' foot.info | sort | uniq -c | sort -nr 266 \E 25 \: 16 \\ 2 \n 1 \r 1 ^I 1 ^H 1 ^G 1 ^? ``` [`terminfo(5)`]: https://man7.org/linux/man-pages/man5/terminfo.5.html#:~:text=Types%20of%20Capabilities [unescape_char]: https://gitlab.com/craigbarnes/lua-terminfo-parser/-/blob/0ff4193563befb8e7c4a0530d277d1b549a2e4ea/terminfo-parser.lua#L21-34

craigbarnes commented 2025年01月25日 20:27:05 +01:00

Just dropping this here for context, now that the Ghostty repo is public: https://github.com/ghostty-org/ghostty/issues/1699

Just dropping this here for context, now that the Ghostty repo is public: https://github.com/ghostty-org/ghostty/issues/1699