Codeberg/Community
54
325
Fork
You've already forked Community
12

429: Too many requests #1645

Open
opened 2024年09月06日 15:11:48 +02:00 by ethical_haquer · 23 comments

Comment

I'm experimenting with mirroring Codeberg issues with GitHub, and vise-versa.
However, Codeberg returned a "429: Too many requests" error.
You can see the GitHub Actions workflow I'm using to sync the GH issues to CB here.
I feel like this is a little too strict, as the GitHub to Codeberg Issue sync workflow had only run 13 times in 18 hours.
Here are the two repos (based off https://codeberg.org/Tuxilio/sync-test):
Codeberg
GitHub
TIA.

### Comment I'm experimenting with mirroring Codeberg issues with GitHub, and vise-versa. However, Codeberg returned a "429: Too many requests" error. You can see the GitHub Actions workflow I'm using to sync the GH issues to CB [here](https://codeberg.org/ethical_haquer/mirror-test/src/branch/main/.github/workflows/issue-sync_github-codeberg.yml). I feel like this is a little too strict, as the GitHub to Codeberg Issue sync workflow had only run 13 times in 18 hours. Here are the two repos (based off https://codeberg.org/Tuxilio/sync-test): [Codeberg](https://codeberg.org/ethical_haquer/mirror-test) [GitHub](https://github.com/ethical-haquer/mirror-test) TIA.
Owner
Copy link

We have adjusted rate-limiting w.r.t. to Forgejo Actions recently. Does the issue still occur?

We have adjusted rate-limiting w.r.t. to Forgejo Actions recently. Does the issue still occur?

@fnetX
Thanks for the reply!
Yeah, it still occurs. Today, I tried again for the first time after opening this issue - The very first run I got a 429:

Issue body: <p><table><tr><th><details><summary></summary>This is an issue created by automatic synchronization.</details></th><th><a href=https://codeberg.org/ethical_haquer><img src=https://codeberg.org/avatars/ce3266a547075f8a012a48ab6b29717b7eedd429dbc86e0bab2c4ea38f6a6cc0 width=25></a></th><th><a href=https://codeberg.org/ethical_haquer>ethical_haquer</a> commented on Codeberg</th><th><a href=https://codeberg.org/ethical_haquer/mirror-test/issues/15>Link to the issue</a></th></tr></table></p>Hello GH!
Comment body: Hello CB!
Event Name: issue_comment
Creating Issue Comment
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2706 0 2224 100 482 3663 793 --:--:-- --:--:-- --:--:-- 4457
100 2706 0 2224 100 482 3663 793 --:--:-- --:--:-- --:--:-- 4457
<!DOCTYPE html>
<html class="codeberg-design">
<head>
 <meta charset="utf-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
 <meta name="viewport" content="width=device-width" />
 <title>429: Too many requests</title>
 
 <link rel="icon" href="https://design.codeberg.org/logo-kit/favicon.ico" type="image/x-icon" />
 <link rel="icon" href="https://design.codeberg.org/logo-kit/favicon.svg" type="image/svg+xml" />
 <link rel="apple-touch-icon" href="https://design.codeberg.org/logo-kit/apple-touch-icon.png" />
 <link rel="stylesheet" href="https://design.codeberg.org/design-kit/codeberg.css" />
 <script defer src="https://design.codeberg.org/design-kit/codeberg.js"></script>
 <script defer type="module" src="https://design.codeberg.org/components/codeberg-components.js"></script>
 <link href="https://fonts.codeberg.org/dist/inter/Inter%20Web/inter.css" rel="stylesheet" />
 <link href="https://fonts.codeberg.org/dist/fontawesome5/css/all.min.css" rel="stylesheet" />
</head>
<body>
 <div class="page-wrapper d-flex align-items-center"> 
 <div class="content">
 <h1>:(</h1>
 <h2 class="content-title">
 You sent too many requests to Codeberg.
 </h2>
 <p>
 We are sorry for the inconvenience, but the wild west situation around the AI gold rush has prompted many big players to "simply crawl the web". Their massive resources can cause traffic that is much more than Codeberg can handle. It was a difficult decision for us, but we started to block suspicious traffic more aggressively to keep Codeberg available and performant to most of our users. And your traffic was also suspicious to us.
 </p>
 <p>
 If you run into this error for the first time, wait about half an hour and try again. If you keep running into this with your normal usage, please contact us so that we can improve our rate limiting. Write to us: <a href="mailto:codeberg@codeberg.org">email codeberg@codeberg.org</a>, <a href="https://social.anoxinon.de/@codeberg">Mastodon</a> or our <a href="https://matrix.to/#/#codeberg.org:matrix.org">Matrix channel</a>.
 </p>
 <p>Thank you for your understanding.</p>
 </div>
 </div>
</body>
</html>

Like I said, here is the GH action I'm using to push issues (and issue comments) on GH to Codeberg, as they're opened. Let me know if you have any questions!

@fnetX Thanks for the reply! Yeah, it still occurs. Today, I tried again for the first time after opening this issue - The very first run I got a 429: ``` Issue body: <p><table><tr><th><details><summary></summary>This is an issue created by automatic synchronization.</details></th><th><a href=https://codeberg.org/ethical_haquer><img src=https://codeberg.org/avatars/ce3266a547075f8a012a48ab6b29717b7eedd429dbc86e0bab2c4ea38f6a6cc0 width=25></a></th><th><a href=https://codeberg.org/ethical_haquer>ethical_haquer</a> commented on Codeberg</th><th><a href=https://codeberg.org/ethical_haquer/mirror-test/issues/15>Link to the issue</a></th></tr></table></p>Hello GH! Comment body: Hello CB! Event Name: issue_comment Creating Issue Comment % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 2706 0 2224 100 482 3663 793 --:--:-- --:--:-- --:--:-- 4457 100 2706 0 2224 100 482 3663 793 --:--:-- --:--:-- --:--:-- 4457 <!DOCTYPE html> <html class="codeberg-design"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" /> <meta name="viewport" content="width=device-width" /> <title>429: Too many requests</title> <link rel="icon" href="https://design.codeberg.org/logo-kit/favicon.ico" type="image/x-icon" /> <link rel="icon" href="https://design.codeberg.org/logo-kit/favicon.svg" type="image/svg+xml" /> <link rel="apple-touch-icon" href="https://design.codeberg.org/logo-kit/apple-touch-icon.png" /> <link rel="stylesheet" href="https://design.codeberg.org/design-kit/codeberg.css" /> <script defer src="https://design.codeberg.org/design-kit/codeberg.js"></script> <script defer type="module" src="https://design.codeberg.org/components/codeberg-components.js"></script> <link href="https://fonts.codeberg.org/dist/inter/Inter%20Web/inter.css" rel="stylesheet" /> <link href="https://fonts.codeberg.org/dist/fontawesome5/css/all.min.css" rel="stylesheet" /> </head> <body> <div class="page-wrapper d-flex align-items-center"> <div class="content"> <h1>:(</h1> <h2 class="content-title"> You sent too many requests to Codeberg. </h2> <p> We are sorry for the inconvenience, but the wild west situation around the AI gold rush has prompted many big players to "simply crawl the web". Their massive resources can cause traffic that is much more than Codeberg can handle. It was a difficult decision for us, but we started to block suspicious traffic more aggressively to keep Codeberg available and performant to most of our users. And your traffic was also suspicious to us. </p> <p> If you run into this error for the first time, wait about half an hour and try again. If you keep running into this with your normal usage, please contact us so that we can improve our rate limiting. Write to us: <a href="mailto:codeberg@codeberg.org">email codeberg@codeberg.org</a>, <a href="https://social.anoxinon.de/@codeberg">Mastodon</a> or our <a href="https://matrix.to/#/#codeberg.org:matrix.org">Matrix channel</a>. </p> <p>Thank you for your understanding.</p> </div> </div> </body> </html> ``` Like I said, [here](https://github.com/ethical-haquer/mirror-test/blob/main/.github/workflows/issue-sync_github-codeberg.yml) is the GH action I'm using to push issues (and issue comments) on GH to Codeberg, as they're opened. Let me know if you have any questions!

Also, if I open a new GitHub issue, it will be created in Codeberg, it's the issue comments it's not syncing - If I make a new comment on GitHub, Codeberg returns a 429 when I try to mirror it on Codeberg.

Also, if I open a new GitHub issue, it _will_ be created in Codeberg, it's the issue comments it's not syncing - If I make a new comment on GitHub, Codeberg returns a 429 when I try to mirror it on Codeberg.

I just got this exact same error when I wanted to fetch/pull like always via HTTPS. Has the rate limiting been changed lately?

I just got this exact same error when I wanted to fetch/pull like always via HTTPS. Has the rate limiting been changed lately?

I am also experiencing this issue when fetching/pulling via HTTPS from my VPS at a known German hosting company. There seems to be a rate limit / block on it's IP range.

I am also experiencing this issue when fetching/pulling via HTTPS from my VPS at a known German hosting company. There seems to be a rate limit / block on it's IP range.

In the meantime I tried some workarounds to find out what is causing the 429.

I have some PHP packages that I git-pull or install via Composer. Both use HTTPS in the Git protocol.

A "git pull" on my local machine works well. On my Netcup server it does not.
If I add option --ipv4 in a "git pull" all works well too, so I think (like @bindaa) that Codeberg have set a rate limiting or maybe even blacklisted some IPv6 ranges that include my server.

As a workaround on Linux I recommend setting a higher IPv4 preference in the "getaddrinfo config file":

in /etc/gai.conf:

precedence ::ffff:0:0/96 100

This will effectively disable outgoing IPv6 connections, resulting in IPv4-only Git network operations.

In the meantime I tried some workarounds to find out what is causing the 429. I have some PHP packages that I git-pull or install via Composer. Both use HTTPS in the Git protocol. A "git pull" on my local machine works well. On my Netcup server it does not. If I add option --ipv4 in a "git pull" all works well too, so I think (like @bindaa) that Codeberg have set a rate limiting or maybe even blacklisted some IPv6 ranges that include my server. As a workaround on Linux I recommend setting a higher IPv4 preference in the "getaddrinfo config file": in `/etc/gai.conf`: `precedence ::ffff:0:0/96 100` This will effectively disable outgoing IPv6 connections, resulting in IPv4-only Git network operations.

@fnetX Of course I'd rather like the limit/restriction get removed for my server's IPv6 range. Would that be possible? Is there an official way or process to ask for such de-blacklisting?

@fnetX Of course I'd rather like the limit/restriction get removed for my server's IPv6 range. Would that be possible? Is there an official way or process to ask for such de-blacklisting?
Owner
Copy link

@Thalian I am not aware of blocks in the Netcup AS ranges, but you can request an explicit check via https://codeberg.org/Codeberg-e.V./requests just in case.

Normally, permanent blocks only occur for excessive crawling, but the patterns are very distinct from regular git pull/push access. Do you receive a "429 Too many requests" call and for which URL? Does it happen immediately or only with some previous usage?

@Thalian I am not aware of blocks in the Netcup AS ranges, but you can request an explicit check via https://codeberg.org/Codeberg-e.V./requests just in case. Normally, permanent blocks only occur for excessive crawling, but the patterns are very distinct from regular git pull/push access. Do you receive a "429 Too many requests" call and for which URL? Does it happen immediately or only with some previous usage?

I just removed the precedence setting from /etc/gai.conf to check IPv6 again, and now there is no issue anymore.

I just removed the precedence setting from `/etc/gai.conf` to check IPv6 again, and now there is no issue anymore.

Hi,

I'm also experiencing the "429 Too Many Requests" error when trying to clone (but push is also a problem) a repository from Codeberg. I've tried the following without success:

  • Waiting and retrying the command.

  • Connecting through a VPN to test if the issue is related to my IP address or location. (but it does not seem to be the case)

Thanks for any help!
Ps: Also took a look with tcpshark to analyse if some program is producing the issue

update:
Cloning repositorys over a device works, until i sign in to codeberg via the webpage. From then on it produces errors.

`~/workbench 
❯ git clone https://codeberg.org/crowci/crow
Klone nach 'crow'...
remote: Enumerating objects: 86525, done.
remote: Counting objects: 100% (1154/1154), done.
remote: Compressing objects: 100% (259/259), done.
remote: Total 86525 (delta 1039), reused 908 (delta 894), pack-reused 85371 (from 1)
Empfange Objekte: 100% (86525/86525), 108.23 MiB | 5.40 MiB/s, fertig.
Löse Unterschiede auf: 100% (54252/54252), fertig.
~/workbench took 25s 
❯ git clone https://codeberg.org/thanasisn/BBand_LAP
Klone nach 'BBand_LAP'... 
remote: Enumerating objects: 11996, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 11996 (delta 190), reused 173 (delta 173), pack-reused 11790 (from 1)
Empfange Objekte: 100% (11996/11996), 8.97 MiB | 2.04 MiB/s, fertig. 
Löse Unterschiede auf: 100% (6927/6927), fertig. 
 
====================
Turned Vpn of
==================== 
~/workbench took 8s 
❯ git clone https://codeberg.org/dluvian/gitplaza
Klone nach 'gitplaza'...
remote: Enumerating objects: 3065, done.
remote: Counting objects: 100% (3065/3065), done.
remote: Compressing objects: 100% (1958/1958), done.
remote: Total 3065 (delta 2259), reused 1557 (delta 1085), pack-reused 0 (from 0)
Empfange Objekte: 100% (3065/3065), 713.19 KiB | 5.40 MiB/s, fertig.
Löse Unterschiede auf: 100% (2259/2259), fertig.
====================
Cloning a my repo
====================
~/workbench 
❯ git clone https://codeberg.org/commitcompanion/switchdialog
Klone nach 'switchdialog'...
Schwerwiegend: konnte nicht auf 'https://codeberg.org/commitcompanion/switchdialog/' zugreifen: The requested URL returned error: 429
~/workbench 
❯ 
`
Hi, I'm also experiencing the "429 Too Many Requests" error when trying to clone (but push is also a problem) a repository from Codeberg. I've tried the following without success: - Waiting and retrying the command. - Connecting through a VPN to test if the issue is related to my IP address or location. (but it does not seem to be the case) Thanks for any help! Ps: Also took a look with tcpshark to analyse if some program is producing the issue update: Cloning repositorys over a device works, until i sign in to codeberg via the webpage. From then on it produces errors. ``` `~/workbench ❯ git clone https://codeberg.org/crowci/crow Klone nach 'crow'... remote: Enumerating objects: 86525, done. remote: Counting objects: 100% (1154/1154), done. remote: Compressing objects: 100% (259/259), done. remote: Total 86525 (delta 1039), reused 908 (delta 894), pack-reused 85371 (from 1) Empfange Objekte: 100% (86525/86525), 108.23 MiB | 5.40 MiB/s, fertig. Löse Unterschiede auf: 100% (54252/54252), fertig. ~/workbench took 25s ❯ git clone https://codeberg.org/thanasisn/BBand_LAP Klone nach 'BBand_LAP'... remote: Enumerating objects: 11996, done. remote: Counting objects: 100% (206/206), done. remote: Compressing objects: 100% (33/33), done. remote: Total 11996 (delta 190), reused 173 (delta 173), pack-reused 11790 (from 1) Empfange Objekte: 100% (11996/11996), 8.97 MiB | 2.04 MiB/s, fertig. Löse Unterschiede auf: 100% (6927/6927), fertig. ==================== Turned Vpn of ==================== ~/workbench took 8s ❯ git clone https://codeberg.org/dluvian/gitplaza Klone nach 'gitplaza'... remote: Enumerating objects: 3065, done. remote: Counting objects: 100% (3065/3065), done. remote: Compressing objects: 100% (1958/1958), done. remote: Total 3065 (delta 2259), reused 1557 (delta 1085), pack-reused 0 (from 0) Empfange Objekte: 100% (3065/3065), 713.19 KiB | 5.40 MiB/s, fertig. Löse Unterschiede auf: 100% (2259/2259), fertig. ==================== Cloning a my repo ==================== ~/workbench ❯ git clone https://codeberg.org/commitcompanion/switchdialog Klone nach 'switchdialog'... Schwerwiegend: konnte nicht auf 'https://codeberg.org/commitcompanion/switchdialog/' zugreifen: The requested URL returned error: 429 ~/workbench ❯ ` ```

@commitcompanion wrote in #1645 (comment):

I'm also experiencing the "429 Too Many Requests" error when trying to clone (but push is also a problem) a repository from Codeberg. I've tried the following without success:

You only experience this via git operations and not when visiting codeberg.org? Is the repository you're trying this action against public?

@commitcompanion wrote in https://codeberg.org/Codeberg/Community/issues/1645#issuecomment-3124265: > I'm also experiencing the "429 Too Many Requests" error when trying to clone (but push is also a problem) a repository from Codeberg. I've tried the following without success: You only experience this via git operations and not when visiting codeberg.org? Is the repository you're trying this action against public?

Just via git operations and with own repos.
Thank you!
Ps: are there cases where an unwanted software caused such a issue?

Just via git operations and with own repos. Thank you! Ps: are there cases where an unwanted software caused such a issue?

I am currently trying to establish GIT in my computer science class at a secondary school. Since we are working with 5+ workstations causing traffic from the same WAN IP this E429 is regularly (read: most of the time) triggered when anyone in the room is trying to access Codeberg (via the webinterface, e.g. even just logging in). A real bottleneck that renders the platform nearly unusable for the purpose at hand (i.e. not having students register at M$ GitHub or similar data-protection-insensitive sites) and still teaching them how to work in the open-source world.

I am currently trying to establish GIT in my computer science class at a secondary school. Since we are working with 5+ workstations causing traffic from the same WAN IP this E429 is regularly (read: most of the time) triggered when anyone in the room is trying to access Codeberg (via the webinterface, e.g. even just logging in). A real bottleneck that renders the platform nearly unusable for the purpose at hand (i.e. not having students register at M$ GitHub or similar data-protection-insensitive sites) and still teaching them how to work in the open-source world.
Owner
Copy link

The rate-limiting is currently per-area (e.g. browsing commits, logging in etc) and based on a window of 30 minutes. The login route is throttled to prevent massive brute-force attacks and massive registration.

How many people need to log in in your class?

The rate-limiting is currently per-area (e.g. browsing commits, logging in etc) and based on a window of 30 minutes. The login route is throttled to prevent massive brute-force attacks and massive registration. How many people need to log in in your class?

Currently there are seven (me included) since not all groups working on an agile design approach decided to incorporate GIT. For future projects there are up to 22 machines that could be used for "synchronous" access.

Currently there are seven (me included) since not all groups working on an agile design approach decided to incorporate GIT. For future projects there are up to 22 machines that could be used for "synchronous" access.

I get the "429 Too many requests" error running Codeberg actions on a self-hosted Forgejo runner.

I get the "429 Too many requests" error running Codeberg actions on a self-hosted Forgejo runner.

It is fixed when renaming your account, which is not a perfect solution.

It is fixed when renaming your account, which is not a perfect solution.
Owner
Copy link

How would renaming an account be of any help?

How would renaming an account be of any help?

I don't know, but it works.

I don't know, but it works.

Hmm..., renaming din't help here. Any idea how to avoid it? It makes actions nearly useless.

Hmm..., renaming din't help here. Any idea how to avoid it? It makes actions nearly useless.

Having the same issues, too. I use self-hosted Forgejo runners and everything. I fail to see how Codeberg expects people to migrate to them, but make Git workflows such a tedious thing due to VERY strict rate limiting.

Having the same issues, too. I use self-hosted Forgejo runners and everything. I fail to see how Codeberg expects people to migrate to them, but make Git workflows such a tedious thing due to VERY strict rate limiting.

For me it works much, much better now. But I also reduced Log output during the runner build, maybe this also helped.

For me it works much, much better now. But I also reduced Log output during the runner build, maybe this also helped.

I have the same problem with pull request and wiki entries that included a lot of graphics. Maybe it would be possible somehow to count loading a page, that embeds graphics from the pull request, commit, or the repo, just as a single request instead of counting a request for every embedded graphic (like a hundreds). Because that multiplies very quickly.

I have the same problem with pull request and wiki entries that included a lot of graphics. Maybe it would be possible somehow to count loading a page, that embeds graphics from the pull request, commit, or the repo, just as a single request instead of counting a request for every embedded graphic (like a hundreds). Because that multiplies very quickly.
Sign in to join this conversation.
No Branch/Tag specified
main
No results found.
Labels
Clear labels
accessibility

Reduces accessibility and is thus a "bug" for certain user groups on Codeberg.
bug

Something is not working the way it should. Does not concern outages.
bug
infrastructure
Errors evidently caused by infrastructure malfunctions or outages
Codeberg

This issue involves Codeberg's downstream modifications and settings and/or Codeberg's structures.
contributions welcome

Please join the discussion and consider contributing a PR!
docs

No bug, but an improvement to the docs or UI description will help
duplicate

This issue or pull request already exists
enhancement

New feature
infrastructure

Involves changes to the server setups, use `bug/infrastructure` for infrastructure-related user errors.
legal

An issue directly involving legal compliance
licence / ToS

involving questions about the ToS, especially licencing compliance
please chill
we are volunteers
Please consider editing your posts and remember that there is a human on the other side. We get that you are frustrated, but it's harder for us to help you this way.
public relations

Things related to Codeberg's external communication
question

More information is needed
question
user support
This issue contains a clearly stated problem. However, it is not clear whether we have to fix anything on Codeberg's end, but we're helping them fix it and/or find the cause.
s/Forgejo

Related to Forgejo. Please also check Forgejo's issue tracker.
s/Forgejo/migration

Migration related issues in Forgejo
s/Pages

Issues related to the Codeberg Pages feature
s/Weblate

Issue is related to the Weblate instance at https://translate.codeberg.org
s/Woodpecker

Woodpecker CI related issue
security

involves improvements to the sites security
service

Add a new service to the Codeberg ecosystem (instead of implementing into Gitea)
upstream

An open issue or pull request to an upstream repository to fix this issue (partially or completely) exists (i.e. Gitea, Forgejo, etc.)
wontfix

Codeberg's current set of contributors are not planning to spend time on delegating this issue.
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
11 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Codeberg/Community#1645
Reference in a new issue
Codeberg/Community
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?