README with many links to images in repo throttled by "Too many requests" #2092

solarchemist commented 2025年08月18日 12:59:22 +02:00

Comment

I have a repo that lists RSS feeds for a selection of newspapers. Part of its README was a table listing each publication's name, feed, and showing its favicon. To avoid lots of requests to the DuckDuckGo or Google favicon API I opted to pre-fetch the PNG files to the repo itself and let the README simply hyperlink to each favicon: [![ico](assets/news.se.png) Stockholm News](http://news.se).

The above approach worked fine up until a couple of months ago. But since then, opening the repo in the browser almost always runs into Codeberg's "Too many requests" page, and when the README does load many favicons show as broken image links and many other repo operations are likewise throttled (like clicking around the repo's files in the browser).

As soon as I removed all the image links from the README browsing the repo works normally again (insert joke about doctor saying "then stop doing that...").

I am not sure how to work-around this. I did try replacing all the image links with base64-encoded strings (thus removing the need for any requests for images) but turns out code forges in general don't support data URIs in Markdown (it's just silently ignored).

Could the Codeberg "request" blocker be tailored to perhaps allow this kind of linking from README to the repo's own assets? Or is the amount of requests (a few hundred favicons) generated by such a list actually excessive no matter where they originate?

Does anyone have any ideas on how to show favicons in this fashion without pummeling either Codeberg or the common favicon APIs?

### Comment I have a [repo that lists RSS feeds for a selection of newspapers](https://codeberg.org/solarchemist/svenska-tidskrifter-rss). Part of its README was a table listing each publication's name, feed, and showing its favicon. To avoid lots of requests to the DuckDuckGo or Google favicon API I opted to pre-fetch the PNG files to the repo itself and let the README simply hyperlink to each favicon: `[ Stockholm News](http://news.se)`. The above approach worked fine up until a couple of months ago. But since then, opening the repo in the browser almost always runs into [Codeberg's "Too many requests" page](https://codeberg.org/attachments/9331240f-6991-470f-bfa2-acabde9fd17d), and when the README does load many favicons show as broken image links and many other repo operations are likewise throttled (like clicking around the repo's files in the browser). As soon as I removed all the image links from the README browsing the repo works normally again (insert joke about doctor saying "then stop doing that..."). I am not sure how to work-around this. I did try replacing all the image links with base64-encoded strings (thus removing the need for any requests for images) but turns out code forges in general don't support data URIs in Markdown (it's just silently ignored). Could the Codeberg "request" blocker be tailored to perhaps allow this kind of linking from README to the repo's own assets? Or is the amount of requests (a few hundred favicons) generated by such a list actually excessive no matter where they originate? Does anyone have any ideas on how to show favicons in this fashion without pummeling either Codeberg or the common favicon APIs?

Gusted commented 2025年08月19日 23:44:16 +02:00

Do you have a indication on the amount of images? The current ratelimiter is 4k req/30min

Do you have a indication on the amount of images? The current ratelimiter is 4k req/30min

solarchemist commented 2025年08月22日 00:55:07 +02:00

Thanks for getting back and sorry the for slow reply.

README currently contains 273 hyperlinks to individual image favicons, plus as many hyperlinks to a single "RSS feed" icon which should only count as a single request. So 274 requests to different image files in the repo from the README.

To help with testing I have just put back the hyperlinks in the README and pushed a new commit and right away 19 favicon images were blocked (as made apparent by "broken image" icon in the browser, also please see attached screenshot of devtools console).

The majority (17 out of 19) of the blocked favicons are the last ones in the README table, but I am not sure that carries any significance. If I hard refresh the page practically all the image favicons now show as "broken images" (devtools console says 99+ blocked requests).

And if (even before the hard refresh) I try to directly access any of these images (whether listed as blocked or not in the console) I get the "Too many requests" page instead, even if I try it from a different browser, so the block is IP-based, I assume?

An example direct image link:

• https://codeberg.org/solarchemist/svenska-tidskrifter-rss/media/branch/main/assets/www.arbetsvarlden.se.png

And since this is the only visit to this repo from my domestic IP address in more than 24h, we should be well below the rate limiter? That thought made me do another test, now from a mobile hotspot, so that should be a completely different IP address as far as Codeberg is concerned. And I see the exact same blocking behaviour, down to which lines in the table are affected on first load, and then mostly everything blocked on refresh. (Both tests in Firefox 128.13.0esr and the other browser being Chromium 120.0.6099.224).

As a work-around I'm considering moving all the image assets away from the repo to a public folder on another server, but if we are an order of magnitude below the limit I would be curious to find out what's going on. I hope it's easy to reproduce by simply opening the repo: https://codeberg.org/solarchemist/svenska-tidskrifter-rss.

Thanks for getting back and sorry the for slow reply. README currently contains 273 hyperlinks to individual image favicons, plus as many hyperlinks to a single "RSS feed" icon which should only count as a single request. So 274 requests to different image files in the repo from the README. To help with testing I have just put back the hyperlinks in the README and pushed a new commit and right away 19 favicon images were blocked (as made apparent by "broken image" icon in the browser, also please see attached screenshot of devtools console). The majority (17 out of 19) of the blocked favicons are the *last* ones in the README table, but I am not sure that carries any significance. If I hard refresh the page practically all the image favicons now show as "broken images" (devtools console says 99+ blocked requests). And if (even before the hard refresh) I try to directly access *any* of these images (whether listed as blocked or not in the console) I get the "Too many requests" page instead, even if I try it from a different browser, so the block is IP-based, I assume? An example direct image link: + https://codeberg.org/solarchemist/svenska-tidskrifter-rss/media/branch/main/assets/www.arbetsvarlden.se.png And since this is the only visit to this repo from my domestic IP address in more than 24h, we should be well below the rate limiter? That thought made me do another test, now from a mobile hotspot, so that should be a completely different IP address as far as Codeberg is concerned. And I see the exact same blocking behaviour, down to which lines in the table are affected on first load, and then mostly everything blocked on refresh. (Both tests in Firefox 128.13.0esr and the other browser being Chromium 120.0.6099.224). As a work-around I'm considering moving all the image assets away from the repo to a public folder on another server, but if we are an order of magnitude below the limit I would be curious to find out what's going on. I hope it's easy to reproduce by simply opening the repo: https://codeberg.org/solarchemist/svenska-tidskrifter-rss.

gedankenstuecke commented 2025年08月26日 15:39:10 +02:00

With CoMaps we're running in the a similar/the same issue with one of our Wiki pages, which is here: https://codeberg.org/comaps/comaps/wiki/Icons

It has ~430 icons used in CoMaps as POI markers, and folks reliably run into the page not loading, even when not being Codeberg power users 😉

With CoMaps we're running in the a similar/the same issue with one of our Wiki pages, which is here: https://codeberg.org/comaps/comaps/wiki/Icons It has ~430 icons used in CoMaps as POI markers, and folks reliably run into the page not loading, even when not being Codeberg power users 😉

vikiawv commented 2025年08月26日 15:39:21 +02:00

Also this file did likely cause a similar issue: https://codeberg.org/comaps/comaps/wiki/Icons
see:
https://comaps.zulipchat.com/#narrow/channel/492331-.F0.9F.92.AC-Watercooler-.28Off-topic.29/topic//near/536236845

Also this file did likely cause a similar issue: https://codeberg.org/comaps/comaps/wiki/Icons see: https://comaps.zulipchat.com/#narrow/channel/492331-.F0.9F.92.AC-Watercooler-.28Off-topic.29/topic//near/536236845

solarchemist commented 2025年10月08日 00:08:54 +02:00

I have implemented a work-around in my own repo, by simply putting all the assets on a remote server that I control and adjusting all the hyperlinks in the README to point to that remote instead of the Codeberg repo. Technically not very hard at all, but I held out just in case some better idea would come to mind.

But I think I should leave this issue open, as the underlying problem has not been solved, and as seen in comments above affects at least one other repo.

The underlying problem, in my estimation, is that once throttling is enabled (at whatever limit) some repos will inevitably become collateral damage. I am far from a web engineer or whatever it's called, but it seems to me that there is fundamentally no way to differentiate between malicious automated requests and "normal" requests - both must necessarily originate from the user's client and target Codeberg.

On a philosophical note, I think this issue nicely illustrates how LLM agents etc. are sawing off the branch they (and we) are sitting on, or to put it bluntly they are breaking the Web. The problem cannot be solved by individual site owners; the operators of these bots must recognize that they are breaking the very infrastructure they depend on and to adhere to the social contract of the web by respecting robots.txt and the like.

I have implemented a work-around in my own repo, by simply putting all the assets on a remote server that I control and adjusting all the hyperlinks in the README to point to that remote instead of the Codeberg repo. Technically not very hard at all, but I held out just in case some better idea would come to mind. But I think I should leave this issue open, as the underlying problem has not been solved, and as seen in comments above affects at least one other repo. The underlying problem, in my estimation, is that once throttling is enabled (at whatever limit) some repos will inevitably become collateral damage. I am far from a web engineer or whatever it's called, but it seems to me that there is fundamentally no way to differentiate between malicious automated requests and "normal" requests - both must necessarily originate from the user's client and target Codeberg. On a philosophical note, I think this issue nicely illustrates how LLM agents etc. are sawing off the branch they (and we) are sitting on, or to put it bluntly they are breaking the Web. The problem cannot be solved by individual site owners; the operators of these bots must recognize that they are breaking the very infrastructure they depend on and to adhere to the social contract of the web by respecting `robots.txt` and the like.