Numbers don't render on some browsers #445

n commented 2021年05月13日 12:23:16 +02:00

Briefly mentioned in #444, numbers fail to render on some browsers because of emoji fonts. (削除) Segoe UI Emoji seems to be to blame. (削除ここまで) Tested on Tor Browser 10.0.16 (based on Mozilla Firefox 78.10.0esr)

---

ref: https://github.com/go-gitea/gitea/pull/15855

[Briefly mentioned](https://codeberg.org/Codeberg/Community/issues/444#issuecomment-197681) in #444, numbers fail to render on some browsers because of emoji fonts. ~~`Segoe UI Emoji` seems to be to blame.~~ Tested on Tor Browser 10.0.16 (based on Mozilla Firefox 78.10.0esr) --- ref: https://github.com/go-gitea/gitea/pull/15855

Codeberg.org commented 2021年05月13日 12:41:32 +02:00

Can reproduce with Tor 10.0.15 and 1.0.16, too.

Please see https://tor.stackexchange.com/questions/16051/why-tor-browser-doesnt-recognize-many-emojis

The second problem Tor Browser faces, more or less uniquely, is that it is trying to block fingerprinting attacks. Websites can determine available fonts on a system, if they are able to do this then they can use the set of installed fonts as a fingerprint. Available fonts split anonymity sets, since Windows users, OSX users and Linux users will all have different fonts available and any user who has added custom fonts will be even more unqiue and trackable. To solve this problem, Tor Browser ships a limited set of fonts with it and tries to restrict it to only use those fonts, so that users have a consistent set of available fonts across platforms making them harder to track and tell apart.

I think this might be the related (the fact that Tor does not rely on the system fonts). I guess there is no option to only ship webfonts if there is no other font available? Because missing numbers are an issue.
I'm not able to reproduce this on any other website, even those who don't ship webfonts. Maybe there is really one font in the list that causes the issue?

Can reproduce with Tor 10.0.15 and 1.0.16, too. Please see https://tor.stackexchange.com/questions/16051/why-tor-browser-doesnt-recognize-many-emojis > The second problem Tor Browser faces, more or less uniquely, is that it is trying to block fingerprinting attacks. Websites can determine available fonts on a system, if they are able to do this then they can use the set of installed fonts as a fingerprint. Available fonts split anonymity sets, since Windows users, OSX users and Linux users will all have different fonts available and any user who has added custom fonts will be even more unqiue and trackable. To solve this problem, Tor Browser ships a limited set of fonts with it and tries to restrict it to only use those fonts, so that users have a consistent set of available fonts across platforms making them harder to track and tell apart. I think this might be the related (the fact that Tor does not rely on the system fonts). I guess there is no option to only ship webfonts if there is no other font available? Because missing numbers *are* an issue. I'm not able to reproduce this on any other website, even those who don't ship webfonts. Maybe there is really one font in the list that causes the issue?

zeripath commented 2021年05月13日 14:01:13 +02:00

This just doesn't make any sense - why would non emoji numbers be affected?

I think we need to see a screenshot of an actual issue that is occuring.

---

If it's just the font rendering issue because the emoji font isn't being provided, well one option for codeberg is to just edit the custom/header.tmpl to include a <style> that adds (I think this is correct - adjust as necessary): (for 1.14.x)

<style>
 @font-face {
 font-family: 'Noto Color Emoji';
 src: url('/NotoColorEmoji.eot'); /* IE9 Compat Modes */
 src: url('/NotoColorEmoji.eot?#iefix') format('embedded-opentype'), /* IE6-IE8 */
 url('/NotoColorEmoji.woff2') format('woff2'), /* Super Modern Browsers */
 url('/NotoColorEmoji.woff') format('woff'), /* Pretty Modern Browsers */
 url('/NotoColorEmoji.ttf') format('truetype'), /* Chrome, Safari, Android, iOS */
 url('/NotoColorEmoji.svg#svgFontName') format('svg'); /* Legacy iOS */
 }
</style>

and then add the font files to the public/ directory. (for 1.15.x the urls should be /assets/NotoColorEmoji.* - again I think this is correct.)

This just doesn't make any sense - why would non emoji numbers be affected? I think we need to see a screenshot of an actual issue that is occuring. --- If it's just the font rendering issue because the emoji font isn't being provided, well one option for codeberg is to just edit the `custom/header.tmpl` to include a `<style>` that adds (I think this is correct - adjust as necessary): (for 1.14.x) ```html <style> @font-face { font-family: 'Noto Color Emoji'; src: url('/NotoColorEmoji.eot'); /* IE9 Compat Modes */ src: url('/NotoColorEmoji.eot?#iefix') format('embedded-opentype'), /* IE6-IE8 */ url('/NotoColorEmoji.woff2') format('woff2'), /* Super Modern Browsers */ url('/NotoColorEmoji.woff') format('woff'), /* Pretty Modern Browsers */ url('/NotoColorEmoji.ttf') format('truetype'), /* Chrome, Safari, Android, iOS */ url('/NotoColorEmoji.svg#svgFontName') format('svg'); /* Legacy iOS */ } </style> ``` and then add the font files to the `public/` directory. (for 1.15.x the urls should be `/assets/NotoColorEmoji.*` - again I think this is correct.)

n commented 2021年05月13日 14:07:01 +02:00

I think we need to see a screenshot of an actual issue that is occuring.

From #444:

Removing Segoe UI Emoji from the --fonts-emoji seems to fix it. I'm not sure why it affects numbers.

> I think we need to see a screenshot of an actual issue that is occuring. From #444:  Removing `Segoe UI Emoji` from the `--fonts-emoji` seems to fix it. I'm not sure why it affects numbers.

fnetX commented 2021年05月13日 14:24:47 +02:00

Reproducing with Tor 10.0.16, it's not enough to remove Segoe UI Emoji from the --fonts-emoji to get rid of the issue, I have to either completely disable (or empty) the property or remove ,"Twemoji Mozilla" from the end of the property.

Reproducing with Tor 10.0.16, it's not enough to remove Segoe UI Emoji from the `--fonts-emoji` to get rid of the issue, I have to either completely disable (or empty) the property or remove `,"Twemoji Mozilla"` from the end of the property.

zeripath commented 2021年05月13日 14:55:19 +02:00

so on chrome it's possible to check what font is providing glyphs in the inspector.

Is it possible to do this on Tor's browser?

for simplicity it might be best to just have the number in a single paragraph by itself.

I suspect Tor's browser has emoji fonts that have the number glyphs in them to prevent number fallbacks being used to identify the browser but... the emoji fonts are lower in priority than the system-ui font - so are we actually looking at the wrong problem? Is it possible that Tor is not liking the system-ui metafont?

Instead of deleting fonts you could try adding a font instead - add whatever is the default font for the Tor browser to the end of --fonts-proportional instead?

so on chrome it's possible to check what font is providing glyphs in the inspector. Is it possible to do this on Tor's browser? for simplicity it might be best to just have the number in a single paragraph by itself. I suspect Tor's browser has emoji fonts that have the number glyphs in them to prevent number fallbacks being used to identify the browser but... the emoji fonts are lower in priority than the system-ui font - so are we actually looking at the wrong problem? Is it possible that Tor is not liking the `system-ui` metafont? Instead of deleting fonts you could try adding a font instead - add whatever is the default font for the Tor browser to the end of `--fonts-proportional` instead?

fnetX commented 2021年05月13日 15:16:59 +02:00

The computed font-family property of the missing issue count in the upper tab is:

-apple-system, "Segoe UI", system-ui, "Roboto", "Helvetica Neue", "Arial", "Noto Sans", "Liberation Sans", "Apple Color Emoji", "Segoe UI Emoji", "Noto Color Emoji", "Twemoji Mozilla", sans-serif

I added "Arimo" to the --fonts-proportional (which is the default font in Tor Browser) and numbers are visible again.

The computed font-family property of the missing issue count in the upper tab is: ~~~ -apple-system, "Segoe UI", system-ui, "Roboto", "Helvetica Neue", "Arial", "Noto Sans", "Liberation Sans", "Apple Color Emoji", "Segoe UI Emoji", "Noto Color Emoji", "Twemoji Mozilla", sans-serif ~~~ I added "Arimo" to the `--fonts-proportional` (which is the default font in Tor Browser) and numbers are visible again.

fnetX commented 2021年05月13日 15:18:52 +02:00

Oh, I thought Firefox would not expose the rendered font, but it's in an extra "Fonts" tab.
Twemoji Mozilla is the used font to render the numbers.
image

Oh, I thought Firefox would not expose the rendered font, but it's in an extra "Fonts" tab. `Twemoji Mozilla` is the used font to render the numbers. 

zeripath commented 2021年05月13日 15:22:05 +02:00

OK - so we have a winner. It's not the fault of the emoji - but rather the fallback process. Nothing has matched before the emoji font matches.

Would it be possible to check if you add sans-serif to end of the --fonts-proportional if that fixes it too?

OK - so we have a winner. It's not the fault of the emoji - but rather the fallback process. Nothing has matched before the emoji font matches. Would it be possible to check if you add `sans-serif` to end of the `--fonts-proportional` if that fixes it too?

zeripath commented 2021年05月13日 15:23:15 +02:00

it would also be useful to ensure that sans-serif doesn't kill the emoji too.

it would also be useful to ensure that sans-serif doesn't kill the emoji too.

fnetX commented 2021年05月13日 15:24:25 +02:00

Yes, that works here, too.

Yes, that works here, too.

fnetX commented 2021年05月13日 15:25:53 +02:00

it would also be useful to ensure that sans-serif doesn't kill the emoji too.

If I do the change in issue #444, numbers appear and the emojis don't disappear. At least on my system with this browser 😂

> it would also be useful to ensure that sans-serif doesn't kill the emoji too. If I do the change in issue #444, numbers appear and the emojis don't disappear. At least on my system with this browser 😂

zeripath commented 2021年05月13日 15:28:08 +02:00

I think that this implies that --fonts-regular should be:

 --fonts-regular: var(--fonts-override, var(--fonts-proportional)), "Noto Sans", "Liberation Sans", sans-serif, var(--fonts-emoji);

not:

 --fonts-regular: var(--fonts-override, var(--fonts-proportional)), "Noto Sans", "Liberation Sans", var(--fonts-emoji), sans-serif;

as it currently is.

(btw I've literally only just realised quite how annoying the emoji selector thing is when you press ':'. I suspect that there should be a delay of 50ms before displaying that thing. I guess this is what happens when you're not dogfooding enough.)

I think that this implies that `--fonts-regular` should be: ```css --fonts-regular: var(--fonts-override, var(--fonts-proportional)), "Noto Sans", "Liberation Sans", sans-serif, var(--fonts-emoji); ``` not: ```css --fonts-regular: var(--fonts-override, var(--fonts-proportional)), "Noto Sans", "Liberation Sans", var(--fonts-emoji), sans-serif; ``` as it currently is. (btw I've literally only just realised quite how annoying the emoji selector thing is when you press ':'. I suspect that there should be a delay of 50ms before displaying that thing. I guess this is what happens when you're not dogfooding enough.)

zeripath commented 2021年05月13日 15:38:38 +02:00

I've put up a PR on GH for this: https://github.com/go-gitea/gitea/pull/15855

I've put up a PR on GH for this: https://github.com/go-gitea/gitea/pull/15855

fnetX commented 2021年05月13日 15:39:05 +02:00

Yes, your proposal works here.

---

Offtopic / to the emoji picker: I'd kinda say it's completely broken or at least really buggy. Try typing and removing a letter from the search - it just sticks on the screen, even if you remove the :. And sometimes it doesn't appear, sometimes it appears out of the window, try typing

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : 

for example and watch how the position jumps up and down if you're fast enough. It's possible to bring it out of screen this way 😂 ← here it didn't appear after typing :joy, had to remvoe and try again ...
I hoped this might change with the switch to plain textarea?

Yes, your proposal works here. --- Offtopic / to the emoji picker: I'd kinda say it's completely broken or at least really buggy. Try typing and removing a letter from the search - it just sticks on the screen, even if you remove the `:`. And sometimes it doesn't appear, sometimes it appears out of the window, try typing ~~~ : : : : : : : : : : : : : : : : : : : : : : : : : : : : : ~~~ for example and watch how the position jumps up and down if you're fast enough. It's possible to bring it out of screen this way 😂 ← here it didn't appear after typing `:joy`, had to remvoe and try again ... I hoped this might change with the switch to plain textarea?

zeripath commented 2021年05月13日 15:59:37 +02:00

OK a workaround would be to add "Noto Sans" to that style mentioned in #445 (comment)

---

Regarding the emoji picker - I think it definitely needs looking at - the question is where is my time best spent. E.g. I need to write a commit-graph reader that's a bit more frugal with memory in order to speed up the native git backend. (I've got a fix into go-git that will mean that at least it won't read 100Mb files into memory and store them in a cache anymore but I think we still need the native git backend.)

OK a workaround would be to add "Noto Sans" to that style mentioned in https://codeberg.org/Codeberg/Community/issues/445#issuecomment-198131 --- Regarding the emoji picker - I think it definitely needs looking at - the question is where is my time best spent. E.g. I need to write a commit-graph reader that's a bit more frugal with memory in order to speed up the native git backend. (I've got a fix into go-git that will mean that at least it won't read 100Mb files into memory and store them in a cache anymore but I think we still need the native git backend.)

fnetX commented 2021年05月13日 16:05:52 +02:00

I think moving up the default sans-serif font is a much more elegant solution. I think we'll look into cherry-picking or so ... but we'll see.

I wouldn't want to waste my time on that tbh, too.
By the way, what's the reason for all the effort with go-git? I don't really see the point in writing a new git backend, sounds just like a lot of effort and currently results in code duplication because some functions are copied in gogit and nogogit files ... but I'm not that deep in Gitea's development, I was just wondering ...
(I read recent issues like 15707 etc, but I still don't get the general picture of how everything works ... don't bother with an answer if you don't want to take the time now, I'll find out someday hehe)

I think moving up the default sans-serif font is a much more elegant solution. I think we'll look into cherry-picking or so ... but we'll see. I wouldn't want to waste my time on that tbh, too. By the way, what's the reason for all the effort with go-git? I don't really see the point in writing a new git backend, sounds just like a lot of effort and currently results in code duplication because some functions are copied in gogit and nogogit files ... but I'm not that deep in Gitea's development, I was just wondering ... (I read recent issues like [15707](https://github.com/go-gitea/gitea/issues/15707) etc, but I still don't get the general picture of how everything works ... don't bother with an answer if you don't want to take the time now, I'll find out someday hehe)

zeripath commented 2021年05月13日 16:28:44 +02:00

By the way, what's the reason for all the effort with go-git? I don't really see the point in writing a new git backend, sounds just like a lot of effort and currently results in code duplication because some functions are copied in gogit and nogogit files ... but I'm not that deep in Gitea's development, I was just wondering ...

You may have noticed that codebergs memory requirements have dropped about 2/3 since the upgrade to 1.14 - that is primarily because of the work I did in writing/restoring the native git backend.

Go-git is simply not frugal with memory at all - and prior to https://github.com/go-git/go-git/pull/303 as I said above it would happily read in a 100Mb file sticking it in a cache to hang around even when you then don't render it because you read the header and find it is 100Mb. This was a serious enough issue that we felt we had to fix it - as otherwise you could just about kill any Gitea server by strategically naming a large file README.md.

Now, why do we need to keep the go-git backend around: Windows. It turns out that Windows has a severe problem with opening git processes so whilst I've worked hard on a reducing the repeated openings I think they're gonna need go-git for a while.

Why do I need to write a commit-graph reader? Well the algorithm implemented for native git backend for commit-info - the thing that says what commit last touched a file - is rather dumb. I used the dumb algorithm because it appeared fast enough locally but it seems that at least for some repository style it is pathologically bad. That means we need to improve the situation.

So now that we have 303 in go-git should we just move back to go-git? Well... I dunno. I still have worries about its memory use and caching. Another worry I have is how we migrate to SHA256.

Hence why I think we're stuck with two backends at least until one shows its definitely better than the other.

> By the way, what's the reason for all the effort with go-git? I don't really see the point in writing a new git backend, sounds just like a lot of effort and currently results in code duplication because some functions are copied in gogit and nogogit files ... but I'm not that deep in Gitea's development, I was just wondering ... You may have noticed that codebergs memory requirements have dropped about 2/3 since the upgrade to 1.14 - that is primarily because of the work I did in writing/restoring the native git backend. Go-git is simply not frugal with memory at all - and prior to https://github.com/go-git/go-git/pull/303 as I said above it would happily read in a 100Mb file sticking it in a cache to hang around even when you then don't render it because you read the header and find it is 100Mb. This was a serious enough issue that we felt we had to fix it - as otherwise you could just about kill any Gitea server by strategically naming a large file README.md. Now, why do we need to keep the go-git backend around: Windows. It turns out that Windows has a severe problem with opening git processes so whilst I've worked hard on a reducing the repeated openings I think they're gonna need go-git for a while. Why do I need to write a commit-graph reader? Well the algorithm implemented for native git backend for commit-info - the thing that says what commit last touched a file - is rather dumb. I used the dumb algorithm because it appeared fast enough locally but it seems that at least for some repository style it is pathologically bad. That means we need to improve the situation. So now that we have 303 in go-git should we just move back to go-git? Well... I dunno. I still have worries about its memory use and caching. Another worry I have is how we migrate to SHA256. Hence why I think we're stuck with two backends at least until one shows its definitely better than the other.

6543 commented 2021年05月15日 21:06:08 +02:00

do we have an upstream issue for this already?

do we have an upstream issue for this already?

6543 commented 2021年05月15日 21:06:29 +02:00

@fnetX you added the upstream lable ...

@fnetX you added the upstream lable ...

fnetX commented 2021年05月15日 21:07:04 +02:00

We have a PR #445 (comment) (you even reviewed 😛)
But the reasonf or the label was that I fired the combined report for the two font issues and someone came in :')

---

Bug bug bug: 😂

We have a PR https://codeberg.org/Codeberg/Community/issues/445#issuecomment-198194 (you even reviewed 😛)

renders as
We have a PR #445 (comment) (you even reviewed 😛)

Ah it's #212 again ...

We have a PR https://codeberg.org/Codeberg/Community/issues/445#issuecomment-198194 (you even reviewed 😛) But the reasonf or the label was that I fired the combined report for the two font issues and someone came in :') --- Bug bug bug: 😂 ~~~ We have a PR https://codeberg.org/Codeberg/Community/issues/445#issuecomment-198194 (you even reviewed 😛) ~~~ renders as We have a PR https://codeberg.org/Codeberg/Community/issues/445#issuecomment-198194 (you even reviewed 😛) Ah it's #212 again ...

6543 commented 2021年05月15日 21:14:09 +02:00

ah ok didnt notice: "numbers not working" == "browser use emoji font to try to display numbers"

PS: updated this issue desc.

ah ok didnt notice: "numbers not working" == "browser use emoji font to try to display numbers" PS: updated *this* issue desc.

fnetX commented 2021年05月15日 21:22:38 +02:00

Yeah sorry for the confusion, I thought those two issues would be related at first.
And sorry for the spam here, I was a bit clumsy with my PC 🙈

Yeah sorry for the confusion, I thought those two issues would be related at first. And sorry for the spam here, I was a bit clumsy with my PC 🙈

fnetX commented 2021年05月19日 17:53:09 +02:00

ashimokawa deployed the patch. Confirmed working in Tor 10.0.16

ashimokawa deployed the patch. Confirmed working in Tor 10.0.16