HTTP 429 for Luanti ContentDB polling repositories #2113

rubenwardy commented 2025年08月29日 11:34:48 +02:00

Comment

Hi, I'm the developer of ContentDB, a website for games, mods, and textures for Luanti. We hung out with Codeberg at FOSDEM.

ContentDB has a feature where it checks for new commits or tags on upstream repositories once per day, using git ls-remote. ContentDB does also support GitHub/GitLab compatible webhooks, this polling feature exists as a platform-independent way to automatically make new releases on ContentDB. ContentDB has 350 packages with codeberg repositories.

From reading the platform infrastructure file, it looks like the rate limit is 250 per 30 minutes.

I have given codeberg a dedicated task queue with a limit of 400/h. Celery task ratelimits work by dividing the time by the number of requests and using that as a minimum time between tasks. So for the 4 workers, each will wait 60*60/100=36 seconds between checking codeberg updates.

ContentDB is still hitting the ratelimit. I'd appreciate your guidance on this. Is 250/30mins correct? Is git ls-remote resulting in multiple git ops? Does every call require an auth op?

### Comment Hi, I'm the developer of [ContentDB](https://content.luanti.org), a website for games, mods, and textures for [Luanti](https://www.luanti.org). We hung out with Codeberg at FOSDEM. ContentDB has a [feature](https://content.luanti.org/help/update_config/) where it checks for new commits or tags on upstream repositories once per day, using `git ls-remote`. ContentDB does also support GitHub/GitLab compatible webhooks, this polling feature exists as a platform-independent way to automatically make new releases on ContentDB. ContentDB has 350 packages with codeberg repositories. From reading the [platform infrastructure](https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/commit/c3d59d41f87d6a5e833f9d1bb14e06e2d988ccbc/hosts/_reverseproxy/etc/caddy/forgejo-prod.site#L55-L80) file, it looks like the rate limit is 250 per 30 minutes. I have given codeberg a dedicated task queue with a limit of 400/h. Celery task ratelimits work by dividing the time by the number of requests and using that as a minimum time between tasks. So for the 4 workers, each will wait `60*60/100=36` seconds between checking codeberg updates. ContentDB is still hitting the ratelimit. I'd appreciate your guidance on this. Is 250/30mins correct? Is `git ls-remote` resulting in multiple git ops? Does every call require an `auth` op?

fnetX commented 2025年08月29日 11:47:08 +02:00

Hi! Sorry for this issue. We recently modified the rate-limiting on August 22 after Codeberg was heavily cloned from by some unknown actor, but the commit wasn't yet pushed to the public (now it is). It has also been adjusted from time to time after that.

The current config is about 150/30 mins. I think that some Git operations do multiple requests (around 2 or 3 IIRC).

Authentication currently does not matter.

Would this work for you (going down to about 50 updates per 30 min)? We're also slowly adjusting the limits (mostly into relaxing it) and appreciate your feedback. If your IP address is static and can be public, we can also put you on the allowlist if you prefer. It's not an issue and you would not need to do further adjustments.

Hi! Sorry for this issue. We recently [modified the rate-limiting on August 22](https://social.anoxinon.de/@Codeberg/115074265437710727) after Codeberg was heavily cloned from by some unknown actor, but the commit wasn't yet pushed to the public ([now it is](https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/commit/3a49625f3440e9895f565f755a3008cced0f43a1)). It has also been adjusted from time to time after that. The current config is about 150/30 mins. I think that some Git operations do multiple requests (around 2 or 3 IIRC). Authentication currently does not matter. Would this work for you (going down to about 50 updates per 30 min)? We're also slowly adjusting the limits (mostly into relaxing it) and appreciate your feedback. If your IP address is static and can be public, we can also put you on the allowlist if you prefer. It's not an issue and you would not need to do further adjustments.

rubenwardy commented 2025年09月04日 15:08:51 +02:00

Okay, it looks like setting the limit to 100/h fixes the issue. It takes about 3.5 hours to check the 350 repos. It's not a hugely time sensitive operation as long as it's done once per day, but does put a limit of 2,000 codeberg repos.

One problem is that other tasks are currently being blocked whilst the codeberg checks are running, I'll need to work out if it is possible to get other tasks to run in between codeberg checks. Perhaps if there's a way to assign priorities so user-visible tasks run first. But this is very much a me problem, even if the codeberg limit was higher it would still be an issue (just less obvious)

Okay, it looks like setting the limit to 100/h fixes the issue. It takes about 3.5 hours to check the 350 repos. It's not a hugely time sensitive operation as long as it's done once per day, but does put a limit of 2,000 codeberg repos. One problem is that other tasks are currently being blocked whilst the codeberg checks are running, I'll need to work out if it is possible to get other tasks to run in between codeberg checks. Perhaps if there's a way to assign priorities so user-visible tasks run first. But this is very much a me problem, even if the codeberg limit was higher it would still be an issue (just less obvious)

fnetX commented 2025年09月04日 15:40:22 +02:00

The allowlist of IP addresses is public at https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/branch/main/hosts/_reverseproxy/srv/ceph/data/production/caddy/allowed.acl. I'll repeat my offer to add you to that list.

Given the current status of the web, being generous about the allowlist seems like a better strategy then being more generous about default limits. It's sad, but feels like the better procedure going forward.

The allowlist of IP addresses is public at https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/branch/main/hosts/_reverseproxy/srv/ceph/data/production/caddy/allowed.acl. I'll repeat my offer to add you to that list. Given the current status of the web, being generous about the allowlist seems like a better strategy then being more generous about default limits. It's sad, but feels like the better procedure going forward.

rubenwardy commented 2025年09月04日 15:43:19 +02:00

Okay well that would allow ContentDB to run a bit more efficiently so I would gladly accept that. The fixed ip addresses are:

194.36.147.174
2a03:4000:4d:b58:58ba:a4ff:fe58:ede
144.91.102.58
2a02:c207:2245:7375::1

Okay well that would allow ContentDB to run a bit more efficiently so I would gladly accept that. The fixed ip addresses are: ``` 194.36.147.174 2a03:4000:4d:b58:58ba:a4ff:fe58:ede 144.91.102.58 2a02:c207:2245:7375::1 ```

fnetX commented 2025年09月04日 16:01:56 +02:00

It should be live now. Could you check?

It should be live now. Could you check?

pat-s commented 2025年09月11日 23:31:55 +02:00

I now got yet another 429 for the (low-level) agent for Crow CI running only renovate (this is another agent than the one in Codeberg-Infrastructure/scripted-configuration#84).

While I also triggered some manual builds outside of the regular hourly interval, I think the current rate limit is too aggressive for "average" development.

I now got yet another 429 for the (low-level) agent for Crow CI running *only* `renovate` (this is another agent than the one in https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/pulls/84). While I also triggered some manual builds outside of the regular hourly interval, I think the current rate limit is too aggressive for "average" development.

rubenwardy commented 2025年09月15日 13:58:25 +02:00

@fnetX wrote in #2113 (comment):

It should be live now. Could you check?

All appears to be working correctly, thank you

@fnetX wrote in https://codeberg.org/Codeberg/Community/issues/2113#issuecomment-6890533: > It should be live now. Could you check? All appears to be working correctly, thank you