Issues with terminfo in a non-standard location #695

dnkl commented 2021年08月28日 12:16:11 +02:00

This is an attempt to collect all issues seen with having our terminfo installed in a non-standard location:

1. sudo (#691, #694), doas (#692), ssh (#691) all drop TERMINFO by default. All can be configured to not do that, but e.g. ssh requires modifying both the client and server configs. Regardless, it's bad user experience
2. gpg decrypt fails (#697).
3. ncurses can be built with --disable-root-environ
4. Exporting TERMINFO apparently breaks the netbsd-curses build. Not sure that one can be blamed on us though.

One of the reasons I went with the current approach (installing terminfo to a non-standard location, and setting TERMINFO) was to be able gracefully fallback to the ncurses provided terminfo definitions.

Once ncurses with its new foot terminfo definitions are out, point 1 and 2 in the list will be a smaller problem since then we'll fallback to ncurses.

However, it also means we silently switch to a less capable terminfo.

Thus, the question is, do we wait for ncurses to be released, and have that "fix" these issues (by silently falling back to a less capable terminfo), or do we change foot's default terminfo to something different (e.g. fot - "foot" in Swedish) and install it in the standard location, and stop exporting TERMINFO?

This is an attempt to collect all issues seen with having our terminfo installed in a non-standard location: 1. sudo (https://codeberg.org/dnkl/foot/issues/691, https://codeberg.org/dnkl/foot/issues/694), doas (https://codeberg.org/dnkl/foot/issues/692), ssh (https://codeberg.org/dnkl/foot/issues/691) all drop `TERMINFO` by default. All can be configured to not do that, but e.g. ssh requires modifying both the client and server configs. Regardless, it's bad user experience 1. gpg decrypt fails (https://codeberg.org/dnkl/foot/issues/697). 1. ncurses can be built with `--disable-root-environ` 1. Exporting `TERMINFO` apparently breaks the netbsd-curses build. Not sure that one can be blamed on us though. One of the reasons I went with the current approach (installing terminfo to a non-standard location, and setting `TERMINFO`) was to be able gracefully fallback to the ncurses provided terminfo definitions. Once ncurses with its new foot terminfo definitions are out, point 1 and 2 in the list will be a smaller problem since then we'll fallback to ncurses. However, it also means we silently switch to a less capable terminfo. Thus, the question is, do we wait for ncurses to be released, and have that "fix" these issues (by silently falling back to a less capable terminfo), or do we change foot's default terminfo to something different (e.g. `fot` - "foot" in Swedish) and install it in the standard location, and stop exporting `TERMINFO`?

dnkl commented 2021年08月28日 13:19:31 +02:00

Seeing that several distributions are likely to rename the terminfos (and install them in the standard location) anyway, I think it makes sense to make that the default here in upstream as well.

Thus, we need a new name for the terminfo. Name suggestions so far:

• fot ("foot" in Swedish)
• bigfoot (as in, bigger, more featureful than "foot" from ncurses)
• foot-term
• foot-sure
• foot-extra
• foot-directors-cut (no, not being entirely serious about this one...)

Seeing that several distributions are likely to rename the terminfos (and install them in the standard location) anyway, I think it makes sense to make that the default here in upstream as well. Thus, we need a new name for the terminfo. Name suggestions so far: * fot ("foot" in Swedish) * bigfoot (as in, bigger, more featureful than "foot" from ncurses) * foot-term * foot-sure * foot-extra * foot-directors-cut (no, not being entirely serious about this one...)

dnkl commented 2021年08月28日 14:16:20 +02:00

Note the following comment from https://dickey.his.com/ncurses/terminfo.ti.html:

# Terminal names look like <manufacturer> <model> - <modes/options>
# The part to the left of the dash, if a dash is present, describes the
# particular hardware of the terminal. The part to the right may be used
# for flags indicating special ROMs, extra memory, particular terminal modes,
# or user preferences.

Our case is a bit of a gray area. But our terminfo and ncurses' do represent the same "manufacturer" and model, and our terminfo can be considered a "mode" (the "More Features" mode).

Which means I think foot-<something> is ok.

Note the following comment from https://dickey.his.com/ncurses/terminfo.ti.html: > _\# Terminal names look like \<manufacturer> \<model> - \<modes/options> > \# The part to the left of the dash, if a dash is present, describes the > \# particular hardware of the terminal. The part to the right may be used > \# for flags indicating special ROMs, extra memory, particular terminal modes, > \# or user preferences._ Our case is a bit of a gray area. But our terminfo and ncurses' _do_ represent the same "manufacturer" and model, and our terminfo _can_ be considered a "mode" (the "More Features" mode). Which means I think `foot-<something>` is ok.

dogknowsnx commented 2021年08月28日 14:47:43 +02:00

First of, thanks for maintaining this great piece of software.

I copied the terminfo back to the standard location for the time being, and all is working fine so far (had issues with manpages after the update).

How about "foot-sure"?

First of, thanks for maintaining this great piece of software. I copied the terminfo back to the standard location for the time being, and all is working fine so far (had issues with manpages after the update). How about "foot-sure"?

dnkl commented 2021年08月28日 14:52:03 +02:00

I copied the terminfo back to the standard location for the time being, and all is working fine so far.

Copying them to ~/.terminfo/f/ should also work, if you don't want to litter the system directories.

How about "foot-sure"?

I may be too tired, but what's the logic behind it?

> I copied the terminfo back to the standard location for the time being, and all is working fine so far. Copying them to `~/.terminfo/f/` _should_ also work, if you don't want to litter the system directories. > How about "foot-sure"? I may be too tired, but what's the logic behind it?

dogknowsnx commented 2021年08月28日 14:59:44 +02:00

I copied the terminfo back to the standard location for the time being, and all is working fine so far.

Copying them to ~/.terminfo/f/ should also work, if you don't want to litter the system directories.

How about "foot-sure"?

I may be too tired, but what's the logic behind it?

In English it means something like 'to not slip' or 'to walk steadily'.

Will use ~/.terminfo/f/ instead - thank you for the hint. :) EDIT: It does work very well!

> > I copied the terminfo back to the standard location for the time being, and all is working fine so far. > > Copying them to `~/.terminfo/f/` _should_ also work, if you don't want to litter the system directories. > > > How about "foot-sure"? > > I may be too tired, but what's the logic behind it? In English it means something like 'to not slip' or 'to walk steadily'. Will use `~/.terminfo/f/` instead - thank you for the hint. :) EDIT: It *does* work very well!

flewkey commented 2021年08月28日 15:51:31 +02:00

I think that foot-extra would be a good option, albeit a plain one.

I think that `foot-extra` would be a good option, albeit a plain one.

dnkl commented 2021年08月28日 16:18:19 +02:00

Another issue with renaming the terminfo is all applications out there doing strcmp(TERM, "foot"), or similar...

Hopefully, they already do strncmp(TERM, "foot", 4), to handle the already existing foot-direct terminfo.

But I know for sure there are those that don't.

In any case, for this reason, using a foot-something name is better than a completely new name.

Another issue with renaming the terminfo is all applications out there doing `strcmp(TERM, "foot")`, or similar... Hopefully, they already do `strncmp(TERM, "foot", 4)`, to handle the already existing `foot-direct` terminfo. But I know for sure there are those that don't. In any case, for this reason, using a foot-something name is better than a completely new name.

craigbarnes commented 2021年08月28日 22:17:22 +02:00

FWIW, using hyphens in the "root" name goes against the recommendations in the terminfo(5) man page:

Terminal names (except for the last, verbose entry) should be chosen using the following conventions. The particular piece of hardware making up the terminal should have a root name, thus "hp2621". This name should not contain hyphens. Modes that the hardware can be in, or user preferences, should be indicated by appending a hyphen and a mode suffix. Thus, a vt100 in 132-column mode would be vt100-w.

I make use of this naming convention in dte by extracting the value of $TERM, up to the first hyphen, and then using bsearch() to search a list of known "root" names.

There are already a few terminals that don't follow this convention (xterm-kitty, xterm-termite, ms-terminal) and it's not a particularly big deal to add special cases for those that don't, but I feel it'd be a shame if new terminals started ignoring it en masse.

FWIW, using hyphens in the "root" name goes against the recommendations in the `terminfo(5)` man page: > Terminal names (except for the last, verbose entry) should be chosen using the following conventions. The particular piece of hardware making up the terminal should have a root name, thus "hp2621". This name should not contain hyphens. Modes that the hardware can be in, or user preferences, should be indicated by appending a hyphen and a mode suffix. Thus, a vt100 in 132-column mode would be vt100-w. I make use of this naming convention in `dte` by [extracting] the value of `$TERM`, up to the first hyphen, and then using `bsearch()` to search a list of known "root" names. There are already a few terminals that don't follow this convention (`xterm-kitty`, `xterm-termite`, `ms-terminal`) and it's not a particularly big deal to add special cases for those that don't, but I feel it'd be a shame if new terminals started ignoring it en masse. [extracting]: https://gitlab.com/craigbarnes/dte/-/blob/dbc256bc6b0416875df8a6d1dfc73db09200e3e2/src/terminal/terminal.c#L173-186

dnkl commented 2021年08月28日 22:38:56 +02:00

@craigbarnes if we go with, say, foot-extra, I would still consider "foot" to be the root name. Don't you agree?

I do think it would be nice to have a completely different name, to distance us from ncurses. Just not sure I'm prepared to break applications that are already string comparing against "foot".

@craigbarnes if we go with, say, foot-extra, I would still consider "foot" to be the root name. Don't you agree? I do think it would be nice to have a completely different name, to distance us from ncurses. Just not sure I'm prepared to break applications that are already string comparing against "foot".

craigbarnes commented 2021年08月28日 23:08:37 +02:00

if we go with, say, foot-extra, I would still consider "foot" to be the root name.

Yeah, that's actually a good point. foot-extra would still work fine for my purposes, with no changes required.

I guess we could also consider -extra to be something akin to a "mode", so perhaps that doesn't even break the ncurses "modes" convention either.

I do think it would be nice to have a completely different name, to distance us from ncurses.

I've been thinking maybe it would make sense to just embrace the upstream terminfo entry and try to solve the few remaining issues by some other means. The non-standard capabilities aren't even useful to programs that use the curses API, since no curses implementation actually uses them. They're only there to help programs that query the terminfo DB and then use their own tty routines. Those kinds of programs can just as well detect the same info some other way. I think if we present a strong enough case to the tmux maintainers, they'll add support for $COLORTERM (or something similar).

The packaging situation could be handled by just explaining to packagers "install the provided terminfo entry only if your ncurses version is < x". The compile-from-git situation doesn't really suffer from filename clashes, since people doing that will most likely be installing under /usr/local or ~/.local.

> if we go with, say, foot-extra, I would still consider "foot" to be the root name. Yeah, that's actually a good point. `foot-extra` would still work fine for my purposes, with no changes required. I guess we could also consider `-extra` to be something akin to a "mode", so perhaps that doesn't even break the ncurses "modes" convention either. > I do think it would be nice to have a completely different name, to distance us from ncurses. I've been thinking maybe it would make sense to just embrace the upstream terminfo entry and try to solve the few remaining issues by some other means. The non-standard capabilities aren't even useful to programs that use the curses API, since no curses implementation actually uses them. They're only there to help programs that query the terminfo DB and then use their own tty routines. Those kinds of programs can just as well detect the same info some other way. I think if we present a strong enough case to the `tmux` maintainers, they'll add support for `$COLORTERM` (or something similar). The packaging situation could be handled by just explaining to packagers "install the provided terminfo entry only if your ncurses version is < x". The compile-from-git situation doesn't really suffer from filename clashes, since people doing that will most likely be installing under `/usr/local` or `~/.local`.

rumpelsepp commented 2021年08月29日 09:00:59 +02:00

However, it also means we silently switch to a less capable terminfo.

I apologize for my stupid question. Why is fixing the ncurses upstream terminfo not an option? Or is this issue about a workaround until the ncurses updates are shipped on every machine (which takes a long time I figure).

> However, it also means we silently switch to a less capable terminfo. I apologize for my stupid question. Why is fixing the ncurses upstream terminfo not an option? Or is this issue about a workaround until the ncurses updates are shipped on every machine (which takes a long time I figure).

craigbarnes commented 2021年08月29日 09:41:33 +02:00

Why is fixing the ncurses upstream terminfo not an option?

There's nothing to fix as far as the ncurses maintainer is concerned. He won't add the extra (non-standard) capabilities to the upstream terminfo DB because ncurses itself doesn't need or use them.

Or is this issue about a workaround until the ncurses updates are shipped on every machine (which takes a long time I figure).

It's about a bunch of things. Issues #670 and #671 are probably the best thing to read to get an idea of the tradeoffs involved.

Also relevant: IRC log from yesterday.

> Why is fixing the ncurses upstream terminfo not an option? There's nothing to fix as far as the ncurses maintainer is concerned. He won't add the extra (non-standard) capabilities to the upstream terminfo DB because ncurses itself doesn't need or use them. > Or is this issue about a workaround until the ncurses updates are shipped on every machine (which takes a long time I figure). It's about a bunch of things. Issues #670 and #671 are probably the best thing to read to get an idea of the tradeoffs involved. Also relevant: IRC [log](https://libera.irclog.whitequark.org/foot/2021-08-28#1630189484-1630190767;) from yesterday.

rumpelsepp commented 2021年08月29日 10:10:13 +02:00

There's nothing to fix as far as the ncurses maintainer is concerned. He won't add the extra (non-standard) capabilities to the upstream terminfo DB because ncurses itself doesn't need or use them.

Ah okay. Thanks for the explanation.

> There's nothing to fix as far as the ncurses maintainer is concerned. He won't add the extra (non-standard) capabilities to the upstream terminfo DB because ncurses itself doesn't need or use them. Ah okay. Thanks for the explanation.

dnkl commented 2021年08月29日 10:28:03 +02:00

After sleeping on this, I think @craigbarnes is right, and that the best way forward is to give in and just use the ncurses terminfo definitions.

What this actually means is:

• Default TERM continues to be foot
• Default install location will be ${datadir}/terminfo.
• Install instructions will be updated to reflect this, and to suggest our terminfo definitions are only installed if ncurses is "too old".
• I will keep the non-standard capabilities in our terminfos, and will document, perhaps on the Wiki, how users can manually build and install them to ~/.terminfo.
• We should also document, on the Wiki, how to configure tmux to work with ncurses' terminfos.

The Arch AUR packages have already been updated to install the terminfos to /usr/share/terminfo, and foot no longer exports TERMINFO.

After sleeping on this, I think @craigbarnes is right, and that the best way forward is to give in and just use the ncurses terminfo definitions. What this actually means is: * Default `TERM` continues to be `foot` * Default install location will be `${datadir}/terminfo`. * Install instructions will be updated to reflect this, and to suggest our terminfo definitions are only installed if ncurses is "too old". * I will keep the non-standard capabilities in our terminfos, and will document, perhaps on the Wiki, how users can manually build and install them to `~/.terminfo`. * We should also document, on the Wiki, how to configure tmux to work with ncurses' terminfos. The Arch AUR packages have already been updated to install the terminfos to `/usr/share/terminfo`, and foot no longer exports `TERMINFO`.

dnkl commented 2021年08月29日 10:30:38 +02:00

As for why I decided to do this, rather than e.g. renaming our terminfo to foot-extra, is it causes the least amount of problems.

Renaming the files means potentially breaking user configurations. But more importantly, we'll be breaking applications that string compare against foot.

Furthermore, having a new name also means we cannot fallback to ncurses' when our terminfo isn't available.

As for _why_ I decided to do this, rather than e.g. renaming our terminfo to `foot-extra`, is it causes the least amount of problems. Renaming the files means potentially breaking user configurations. But more importantly, we'll be breaking applications that string compare against `foot`. Furthermore, having a new name also means we cannot fallback to ncurses' when our terminfo isn't available.

Arnavion commented 2021年08月29日 12:04:18 +02:00

Could you list what tmux does differently with TERM=xterm-256color vs TERM=foot (ncurses) vs TERM=foot (foot) ? I think on IRC you mentioned neovim too, so if that's also affected then could list that as well?

Basically if you're saying that using tmux in foot functionally requires the user to configure tmux (to add terminal-overrides) / configure foot (to change term to something else that already works with tmux) / build and install foot's custom terminfo manually, then it sounds like an install of foot will not work with tmux out of the box. I would not like such a situation for OpenSUSE, and would lean towards having the OpenSUSE package install the terminfos as foot-extra and make TERM=foot-extra the default.

Could you list what tmux does differently with `TERM=xterm-256color` vs `TERM=foot` (ncurses) vs `TERM=foot` (foot) ? I think on IRC you mentioned neovim too, so if that's also affected then could list that as well? Basically if you're saying that using tmux in foot functionally *requires* the user to configure tmux (to add `terminal-overrides`) / configure foot (to change `term` to something else that already works with tmux) / build and install foot's custom terminfo manually, then it sounds like an install of foot will not work with tmux out of the box. I would not like such a situation for OpenSUSE, and would lean towards having the OpenSUSE package install the terminfos as `foot-extra` and make `TERM=foot-extra` the default.

dnkl commented 2021年08月29日 12:16:30 +02:00

The biggest issue is truecolor support. Tmux "knows" xterm can do it, so if $TERM contains the string "xterm", it enables truecolor.

Tmux also supports the non-standard terminfo capabilities Tc, setrgbf and setrgbb, and when these are found, it uses them for truecolor, regardless of $TERM. These capabilities are only set in "our" terminfo definitions, but not in any ncurses definitions.

At least that's how I have understood things, but I don't use tmux myself.

Basically if you're saying that using tmux in foot functionally requires the user to configure tmux (to add terminal-overrides) / configure foot (to change term to something else that already works with tmux) / build and install foot's custom terminfo manually, then it sounds like an install of foot will not work with tmux out of the box.

That would be correct. @craigbarnes thinks we can push tmux to recognize $COLORTERM, and if so, that would remove the need for these non-standard capabilities.

and would lean towards having the OpenSUSE package install the terminfos as foot-extra and make TERM=foot-extra the default.

I intend to make that possible. Just be aware that there are applications doing strcmp($TERM, "foot") that will break. Unfortunately, I cannot remember which ones.

There's also the Sync capability, the informs applications foot can do "Application Synchronized Updates, which prevents tearing. But, not having it in the terminfo definition won't break anything. And I'm not aware of anyone but tmux using it (to limited effect, at that).

The biggest issue is truecolor support. Tmux "knows" xterm can do it, so if `$TERM` contains the string "xterm", it enables truecolor. Tmux also supports the non-standard terminfo capabilities `Tc`, `setrgbf` and `setrgbb`, and when these are found, it uses them for truecolor, regardless of `$TERM`. These capabilities are only set in "our" terminfo definitions, but not in **any** ncurses definitions. At least that's how I have understood things, but I don't use tmux myself. > Basically if you're saying that using tmux in foot functionally requires the user to configure tmux (to add terminal-overrides) / configure foot (to change term to something else that already works with tmux) / build and install foot's custom terminfo manually, then it sounds like an install of foot will not work with tmux out of the box. That would be correct. @craigbarnes thinks we can push tmux to recognize $COLORTERM, and if so, that would remove the need for these non-standard capabilities. > and would lean towards having the OpenSUSE package install the terminfos as foot-extra and make TERM=foot-extra the default. I intend to make that possible. Just be aware that there **are** applications doing `strcmp($TERM, "foot")` that will break. Unfortunately, I cannot remember which ones. There's also the `Sync` capability, the informs applications foot can do "Application Synchronized Updates, which prevents tearing. But, not having it in the terminfo definition won't break anything. And I'm not aware of anyone but tmux using it (to limited effect, at that).

dnkl commented 2021年08月29日 13:01:30 +02:00

#698 should be ready now. I'm sure the documentation can be improved upon, but the build system has been updated with the new defaults.

https://codeberg.org/dnkl/foot/pulls/698 should be ready now. I'm sure the documentation can be improved upon, but the build system has been updated with the new defaults.

supermarin commented 2022年01月31日 14:16:16 +01:00

Still an issue on nixos/sway. Setting TERM=xterm fixes the problem with gpg

Still an issue on nixos/sway. Setting TERM=xterm fixes the problem with gpg

dnkl commented 2022年01月31日 15:11:43 +01:00

Report this to nixos, since they build foot in a way that sets TERMINFO_DIRS.

Report this to nixos, since they build foot in a way that sets TERMINFO_DIRS.