Use brotli along gzip #69

rdwz commented 2023年01月19日 16:52:44 +01:00

The same point applies with gzip, but if we can change the web server config to use brotli that would be ideal.

Originally posted by @caesar in #61 (comment)

Unfortunately, Brotli is not part of the default NGINX Open Source package.

Could be loaded as a dynamic or static module.

-> Performance: Enable text compression #845

> The same point applies with gzip, but if we can change the web server config to use brotli that would be ideal. > > _Originally posted by @caesar in https://codeberg.org/forgejo/website/issues/61#issuecomment-772675_ Unfortunately, Brotli is not part of the default NGINX Open Source package. Could be loaded as a dynamic or static module. -> [Performance: Enable text compression #845](https://codeberg.org/Codeberg/Community/issues/845)

rdwz commented 2023年01月24日 23:37:32 +01:00

Re. Brotli... HAProxy doesn't support brotli (and for a very good reason), as it is comparatively memory intensive (~6-10x gzip, which is already ~6x to deflate), this causes exceptionally high memory utilization for compression, which is esp. bad on high traffic reverse proxies / load balancers (HAProxy is made to handle humongous amounts of traffic on a small resource budget). Brotli shines on pre-compressing static assets, but doesn't perform well in inline compression, so adopting it should not be considered on dynamic assets, no matter what... by @gapodo

With Astro this can be done easily

Simple: Set it and forget it
Optimal: By compressing ahead of time, a more performant compression can be performed

https://github.com/sondr3/astro-compressor

> Re. Brotli... HAProxy doesn't support brotli (and for a very good reason), as it is comparatively memory intensive (~6-10x gzip, which is already ~6x to deflate), this causes exceptionally high memory utilization for compression, which is esp. bad on high traffic reverse proxies / load balancers (HAProxy is made to handle humongous amounts of traffic on a small resource budget). Brotli shines on pre-compressing static assets, but doesn't perform well in inline compression, so adopting it should not be considered on dynamic assets, no matter what... by @gapodo With Astro this can be done easily > Simple: Set it and forget it > Optimal: By compressing ahead of time, a more performant compression can be performed > > https://github.com/sondr3/astro-compressor

crystal commented 2023年01月25日 04:17:22 +01:00

I'm not positive about how it's set up, but it really looks like Forgejo.org is currently hosted by a shared hosting server. If this is true, simply adding the brotli module may not be an option.

I'm not positive about how it's set up, but it really looks like Forgejo.org is currently hosted by a shared hosting server. If this is true, simply adding the brotli module may not be an option.

Ghost commented 2023年01月25日 10:09:20 +01:00

This is correct, the pages are uploaded to a hosting service with just static pages. For the sake of maintainability I'd rather not add to the complexity unless there is a compelling reason to do so.

@crystal given the recent tensions I would be very grateful if you could refrain from commenting further on this issue. If you think something important needs to be raised, could you please reach out to me privately? I will convey your message.

This is correct, the pages are uploaded to a hosting service with just static pages. For the sake of maintainability I'd rather not add to the complexity unless there is a compelling reason to do so. @crystal given the recent tensions I would be very grateful if you could refrain from commenting further on this issue. If you think something important needs to be raised, could you please reach out to me privately? I will convey your message.

Ghost commented 2023年01月25日 10:13:11 +01:00

@redwerkz currently nothing specific is done with regard to compression on the host that runs https://forgejo.org. If something (however small) has to be added, it will need maintenance (however minimal) and I am very careful to avoid technical debt unless there is a clear benefit that outweigh the associated recurring work.

Could you please layout for me to costs and benefits of the suggested change?

@redwerkz currently nothing specific is done with regard to compression on the host that runs https://forgejo.org. If something (however small) has to be added, it will need maintenance (however minimal) and I am very careful to avoid technical debt unless there is a clear benefit that outweigh the associated recurring work. Could you please layout for me to costs and benefits of the suggested change?

fsologureng commented 2023年01月25日 13:06:33 +01:00

I deleted a comment which has no addings to the issue. Sorry @redwerkz

I deleted a comment which has no addings to the issue. Sorry @redwerkz

Ghost commented 2023年01月25日 13:09:11 +01:00

The forgejo preview website runs over codeberg pages which runs behind haproxy

It is actually hosted at uberspace.de and I proposed an addition to the README as a reminder #79

> The forgejo preview website runs over codeberg pages which runs behind haproxy It is actually hosted at uberspace.de and I proposed an addition to the README as a reminder https://codeberg.org/forgejo/website/pulls/79

rdwz commented 2023年01月25日 13:46:03 +01:00

as from the user side, they are ready and everybody would benefit. worldwide support is as high as 95.38%, but gzip will remain as fallback anyways https://caniuse.com/brotli

in short with Brotli filesizes can be up to 30% smaller than Gzip'ed, and our static website can fully profit from precompressed assets so the server still remains with full computing power..

He is a datailed article I found: https://csswizardry.com/2020/04/real-world-effectiveness-of-brotli/

the conclusion is this

So What?
It would seem that, realistically, the benefits of Brotli over Gzip are slight.

If enabling Brotli is as simple as flicking a checkbox in the admin panel of your CDN, please go ahead and do it right now: support is wide enough, fallbacks are simple, and even minimal improvements are better than none at all.

Where possible, upload precompressed static assets to your web server with the highest possible compression setting, and use something more middle-ground for anything dynamic. If you’re running on Nginx, please ensure you aren’t still on their pitiful default compression level of 1.

we running that as far as i know

and besides that the overall size of our website can already be reduced by 15% #61 (comment)

as from the user side, they are ready and everybody would benefit. worldwide support is as high as 95.38%, but gzip will remain as fallback anyways https://caniuse.com/brotli in short with Brotli filesizes can be up to 30% smaller than Gzip'ed, and our static website can fully profit from precompressed assets so the server still remains with full computing power.. He is a datailed article I found: https://csswizardry.com/2020/04/real-world-effectiveness-of-brotli/ the conclusion is this > So What? > It would seem that, realistically, the benefits of Brotli over Gzip are slight. > > If enabling Brotli is as simple as flicking a checkbox in the admin panel of your CDN, please go ahead and do it right now: support is wide enough, fallbacks are simple, and even minimal improvements are better than none at all. > > Where possible, upload precompressed static assets to your web server with the highest possible compression setting, and use something more middle-ground for anything dynamic. **If you’re running on Nginx, please ensure you aren’t still on their pitiful default compression level of 1.** we running **that** as far as i know and besides that the overall size of our website can already be reduced by 15% https://codeberg.org/forgejo/website/issues/61#issuecomment-772675

fsologureng commented 2023年01月25日 15:54:29 +01:00

as from the user side, they are ready and everybody would benefit. worldwide support is as high as 95.38%, but gzip will remain as fallback anyways https://caniuse.com/brotli

in short with Brotli filesizes can be up to 30% smaller than Gzip'ed, and our static website can fully profit from precompressed assets so the server still remains with full computing power..

The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used.

He is a datailed article I found: https://csswizardry.com/2020/04/real-world-effectiveness-of-brotli/

the conclusion is this

So What?
It would seem that, realistically, the benefits of Brotli over Gzip are slight.

If enabling Brotli is as simple as flicking a checkbox in the admin panel of your CDN, please go ahead and do it right now: support is wide enough, fallbacks are simple, and even minimal improvements are better than none at all.

Where possible, upload precompressed static assets to your web server with the highest possible compression setting, and use something more middle-ground for anything dynamic. If you’re running on Nginx, please ensure you aren’t still on their pitiful default compression level of 1.

we running that as far as i know

and besides that the overall size of our website can already be reduced by 15%

That measure is for static compression including images, and furthermore: (cite) *Note: astro-compress will not compress your requests, only your statically generated build and pre-rendered, so brotli is not related with that 15%.

All request of images (the main weight of practically any site) will be previously compressed and are always excluded of http compression¹ because is expensive and useless, so adding brotli will produce a little benefit just for text assets but an additional effort of CPU for both server and browsers.

I think our bandwith saving and speed response increase won't compensate additional CPU wasting and maintenance need.

> as from the user side, they are ready and everybody would benefit. worldwide support is as high as 95.38%, but gzip will remain as fallback anyways https://caniuse.com/brotli > > in short with Brotli filesizes can be up to 30% smaller than Gzip'ed, and our static website can fully profit from precompressed assets so the server still remains with full computing power.. The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used. > > He is a datailed article I found: https://csswizardry.com/2020/04/real-world-effectiveness-of-brotli/ > > the conclusion is this > > > So What? > > It would seem that, realistically, the benefits of Brotli over Gzip are slight. > > > > If enabling Brotli is as simple as flicking a checkbox in the admin panel of your CDN, please go ahead and do it right now: support is wide enough, fallbacks are simple, and even minimal improvements are better than none at all. > > > > Where possible, upload precompressed static assets to your web server with the highest possible compression setting, and use something more middle-ground for anything dynamic. **If you’re running on Nginx, please ensure you aren’t still on their pitiful default compression level of 1.** > > we running **that** as far as i know > > and besides that the overall size of our website can already be reduced by 15% That measure is for static compression **including images**, and furthermore: (cite) [*Note: astro-compress will not compress your requests, only your statically generated build and pre-rendered](https://github.com/astro-community/astro-compress#astro-compress-%EF%B8%8F), so brotli is not related with that 15%. All request of images (the main weight of practically any site) will be previously compressed and are always excluded of http compression^{[^1]} because is expensive and useless, so adding brotli will produce a little benefit just for text assets but an additional effort of CPU for both server and browsers. I think our bandwith saving and speed response increase won't compensate additional CPU wasting and maintenance need. [^1]: Except SVG which is a markup text, that's why is good to use it instead of png or webp

rdwz commented 2023年01月25日 17:43:20 +01:00

The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used.

Saving bandwidth in the long run without additional load on each visitor/request.

so brotli is not related with that 15%.

The idea is to get rid of data not needed first, then compress only that in a second step.

> The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used. Saving bandwidth in the long run without additional load on each visitor/request. > so brotli is not related with that 15%. The idea is to get rid of data not needed first, then compress only that in a second step.

Ghost commented 2023年01月25日 18:12:19 +01:00

@redwerkz what needs to be done for brotli to be used at https://uberspace.de/? Could you give me a detailed HOWTO of the technical steps to follow?

@redwerkz what needs to be done for brotli to be used at https://uberspace.de/? Could you give me a detailed HOWTO of the technical steps to follow?

fsologureng commented 2023年01月25日 18:34:03 +01:00

@redwerkz

The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used.

Saving bandwidth in the long run without additional load on each visitor/request.

Brotli compress each request, why you say don't?

@redwerkz >> The precompressed assets of this site don't use brotli as format, so it will be an increase in computing power used. > >Saving bandwidth in the long run without additional load on each visitor/request. Brotli compress each request, why you say don't?

rdwz commented 2023年01月29日 19:05:16 +01:00

In general you are right, but this discussion was based on

Re. Brotli... HAProxy doesn't support brotli (and for a very good reason), as it is comparatively memory intensive (~6-10x gzip, which is already ~6x to deflate), this causes exceptionally high memory utilization for compression, which is esp. bad on high traffic reverse proxies / load balancers (HAProxy is made to handle humongous amounts of traffic on a small resource budget). Brotli shines on pre-compressing static assets, but doesn't perform well in inline compression, so adopting it should not be considered on dynamic assets, no matter what... by @gapodo

@dachary

I will sort this out and get back to this later

In general you are right, but this discussion was based on > Re. Brotli... HAProxy doesn't support brotli (and for a very good reason), as it is comparatively memory intensive (~6-10x gzip, which is already ~6x to deflate), this causes exceptionally high memory utilization for compression, which is esp. bad on high traffic reverse proxies / load balancers (HAProxy is made to handle humongous amounts of traffic on a small resource budget). Brotli shines on pre-compressing static assets, but doesn't perform well in inline compression, so adopting it should not be considered on dynamic assets, no matter what... by @gapodo @dachary I will sort this out and get back to this later

Ryuno-Ki commented 2023年01月30日 15:19:50 +01:00

@redwerkz what needs to be done for brotli to be used at https://uberspace.de/? Could you give me a detailed HOWTO of the technical steps to follow?

https://manual.uberspace.de/web-headers/

Or drop them an email address / reach out to them on Mastodon.

> @redwerkz what needs to be done for brotli to be used at https://uberspace.de/? Could you give me a detailed HOWTO of the technical steps to follow? https://manual.uberspace.de/web-headers/ Or drop them an email address / reach out to them on Mastodon.

rdwz commented 2023年01月30日 16:22:52 +01:00

I tried to have a working poc done with caddy, it should be possible, yet i struggle to config that thing.

Maybe i try another day. And even if manage to make it work, i doubt uberspace will have this custom nginx enrolled and i myself have a clue, if that's worth or good for production use (well, pay-for-nginx does support it 😅) nor do i see this feat. alone to justify a nginx replacment.

My low-effort idea was to minimize assets at build-time, this is just a whataboutism follow up to the average answer of ze "wont fix, does it matter?"-attitude.

Offtopic: uberspace despite the name doesn't seem to live that latest feature game (at all!?).. yes, they can be easily contacted via twitter, but still i'm waiting for actions: like the forgejo guide i wrote for the uberspave lab (2w ago), gpg is outdated -> answer: they opened a internal ticket, latest node 18 isnt support, but eol v12 is still available.. 🫣

I tried to have a working poc done with [caddy](https://caddyserver.com/), it should be possible, yet i struggle to config that thing. Maybe i try another day. And even if manage to make it work, i doubt uberspace will have this custom nginx enrolled and i myself have a clue, if that's worth or good for production use (well, pay-for-nginx does support it 😅) nor do i see this feat. alone to justify a nginx replacment. My low-effort idea was to minimize assets at build-time, this is just a `whataboutism` follow up to the average answer of ze "wont fix, does it matter?"-attitude. Offtopic: uberspace despite the name doesn't seem to live that latest feature game (at all!?).. yes, they can be easily contacted via twitter, but still i'm waiting for actions: like the forgejo guide i wrote for the uberspave lab (2w ago), gpg is outdated -> answer: they opened a internal ticket, latest node 18 isnt support, but eol v12 is still available.. 🫣