Suspected IP shadowban #2133

ieure commented 2025年09月12日 02:08:35 +02:00

Comment

Hi, I'm reaching out to determine if an IP has been shadowbanned. I have two machines, each at different physical locations, each with different Internet connections. One has no issues at all, while the other receives 429 responses for 100% of its requests. I noticed this probably two weeks ago, and the situation has persisted.

Machine A:

$ date --utc && git clone --shallow-since=1d https://codeberg.org/guix/guix.git
Fri Sep 12 12:05:14 AM UTC 2025
Cloning into 'guix'...
remote: Enumerating objects: 5795, done. 
remote: Counting objects: 100% (5795/5795), done. 
remote: Compressing objects: 100% (4006/4006), done. 
remote: Total 5795 (delta 2504), reused 4172 (delta 1766), pack-reused 0 (from 0) 
Receiving objects: 100% (5795/5795), 27.32 MiB | 5.95 MiB/s, done.
Resolving deltas: 100% (2504/2504), done.
Updating files: 100% (3225/3225), done.

Machine B:

$ date --utc && git clone --shallow-since=1d https://codeberg.org/guix/guix.git
Fri Sep 12 12:05:18 AM UTC 2025
Cloning into 'guix'...
fatal: unable to access 'https://codeberg.org/guix/guix.git/': The requested URL returned error: 429

The only difference is the source network the traffic originates from.

How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it?

### Comment Hi, I'm reaching out to determine if an IP has been shadowbanned. I have two machines, each at different physical locations, each with different Internet connections. One has no issues at all, while the other receives 429 responses for 100% of its requests. I noticed this probably two weeks ago, and the situation has persisted. Machine A: ``` $ date --utc && git clone --shallow-since=1d https://codeberg.org/guix/guix.git Fri Sep 12 12:05:14 AM UTC 2025 Cloning into 'guix'... remote: Enumerating objects: 5795, done. remote: Counting objects: 100% (5795/5795), done. remote: Compressing objects: 100% (4006/4006), done. remote: Total 5795 (delta 2504), reused 4172 (delta 1766), pack-reused 0 (from 0) Receiving objects: 100% (5795/5795), 27.32 MiB | 5.95 MiB/s, done. Resolving deltas: 100% (2504/2504), done. Updating files: 100% (3225/3225), done. ``` Machine B: ``` $ date --utc && git clone --shallow-since=1d https://codeberg.org/guix/guix.git Fri Sep 12 12:05:18 AM UTC 2025 Cloning into 'guix'... fatal: unable to access 'https://codeberg.org/guix/guix.git/': The requested URL returned error: 429 ``` The only difference is the source network the traffic originates from. How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it?

Gusted commented 2025年09月12日 15:31:54 +02:00

@ieure wrote in #2133 (comment):

How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it?

Hi, could you please sent your IP to gusted at codeberg dot org?

@ieure wrote in https://codeberg.org/Codeberg/Community/issues/2133#issue-2351617: > How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it? Hi, could you please sent your IP to `gusted at codeberg dot org`?

ieure commented 2025年09月13日 01:11:17 +02:00

@Gusted wrote in #2133 (comment):

@ieure wrote in #2133 (comment):

How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it?

Hi, could you please sent your IP to gusted at codeberg dot org?

Done!

@Gusted wrote in https://codeberg.org/Codeberg/Community/issues/2133#issuecomment-7136632: > @ieure wrote in #2133 (comment): > > > How can I determine if I'm under an IP ban, and if so, the reason for it? How can I appeal it? > > Hi, could you please sent your IP to `gusted at codeberg dot org`? Done!

Gusted commented 2025年09月13日 17:25:56 +02:00

I've looked at the logs and the IP is not on our banlist, however the server logs do indicate that this specific IP address is making git requests quite often to several guix repositories (several times per minute and likely more often but limited by the rate limiter). So you are seeing the rate limiter that is limiting the amount of git requests you can make. Is this intentional?

I've looked at the logs and the IP is not on our banlist, however the server logs do indicate that this specific IP address is making git requests quite often to several guix repositories (several times per minute and likely more often but limited by the rate limiter). So you are seeing the rate limiter that is limiting the amount of git requests you can make. Is this intentional?

ieure commented 2025年09月13日 19:06:27 +02:00

@Gusted wrote in #2133 (comment):

I've looked at the logs and the IP is not on our banlist, however the server logs do indicate that this specific IP address is making git requests quite often to several guix repositories (several times per minute and likely more often but limited by the rate limiter). So you are seeing the rate limiter that is limiting the amount of git requests you can make. Is this intentional?

That machine hosts my Cuirass server, which is a CI tool for Guix. It polls specified repos to look for new things to build, but shouldn't be polling multiple times a minute. The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute.

What's the actual rate I need to stay under?

@Gusted wrote in https://codeberg.org/Codeberg/Community/issues/2133#issuecomment-7152142: > I've looked at the logs and the IP is not on our banlist, however the server logs do indicate that this specific IP address is making git requests quite often to several guix repositories (several times per minute and likely more often but limited by the rate limiter). So you are seeing the rate limiter that is limiting the amount of git requests you can make. Is this intentional? That machine hosts my Cuirass server, which is a CI tool for Guix. It polls specified repos to look for new things to build, but shouldn't be polling multiple times a minute. The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute. What's the actual rate I need to stay under?

Gusted commented 2025年09月13日 21:30:38 +02:00

@ieure wrote in #2133 (comment):

The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute.

What URL does it poll? In the server logs this seems to be doing a git fetch (libgit2 is mentioned in the user-agent), which means it gets the rate limiter for git requests which is quite strict at the moment:

@ieure wrote in https://codeberg.org/Codeberg/Community/issues/2133#issuecomment-7153045: > The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute. What URL does it poll? In the server logs this seems to be doing a git fetch (libgit2 is mentioned in the user-agent), which means it gets the rate limiter for git requests which is quite strict at the moment: https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/commit/ee1c5cfad621c4f66386886fbc822fdabb6c1776/hosts/_reverseproxy/etc/caddy/forgejo-prod.site#L67-L72

ieure commented 2025年09月15日 01:22:14 +02:00

@Gusted wrote in #2133 (comment):

@ieure wrote in #2133 (comment):

The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute.

What URL does it poll? In the server logs this seems to be doing a git fetch (libgit2 is mentioned in the user-agent), which means it gets the rate limiter for git requests which is quite strict at the moment:
Codeberg-Infrastructure/scripted-configuration – hosts/_reverseproxy/etc/caddy/forgejo-prod.site
Lines 67 to 72 in Codeberg-Infrastructure/scripted-configuration@ee1c5cf
zone gitop {
match header User-Agent git/*
key gitop-{client_ip}
window 30m
events 150
}

It's going to be pulling new commits from https://codeberg.org/guix/guix.git and https://codeberg.org/ieure/atomized-guix.git.

150 events per 30 minutes works out to 5/minute, which feels like it should be plenty. I disabled some of the jobs I had running and ought to be safely under that.

I suspect that Cuirass isn't optimizing for the (very common) case where the same commits are needed for multiple jobs, so it's unnecessarily refetching.

@Gusted wrote in https://codeberg.org/Codeberg/Community/issues/2133#issuecomment-7154191: > @ieure wrote in #2133 (comment): > > > The default configuration polls the repos to look for changes to build every five minutes. Since I have five different things I have it set up to build, I suppose this does work out to >1 per minute, though I'd expect it's on the order of 2-3 requests per minute. > > What URL does it poll? In the server logs this seems to be doing a git fetch (libgit2 is mentioned in the user-agent), which means it gets the rate limiter for git requests which is quite strict at the moment: > [Codeberg-Infrastructure/scripted-configuration](https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/) – [hosts/_reverseproxy/etc/caddy/forgejo-prod.site](https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/commit/ee1c5cfad621c4f66386886fbc822fdabb6c1776/hosts/_reverseproxy/etc/caddy/forgejo-prod.site#L67-L72) > Lines 67 to 72 in [Codeberg-Infrastructure/scripted-configuration@ee1c5cf](https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/commit/ee1c5cfad621c4f66386886fbc822fdabb6c1776) > ` zone gitop { ` > ` match header User-Agent git/* ` > ` key gitop-{client_ip} ` > ` window 30m ` > ` events 150 ` > ` } ` It's going to be pulling new commits from `https://codeberg.org/guix/guix.git` and `https://codeberg.org/ieure/atomized-guix.git`. 150 events per 30 minutes works out to 5/minute, which feels like it should be plenty. I disabled some of the jobs I had running and ought to be safely under that. I suspect that Cuirass isn't optimizing for the (very common) case where the same commits are needed for multiple jobs, so it's unnecessarily refetching.

Gusted commented 2025年09月21日 01:02:24 +02:00

We deployed two improvements to the Git rate limiter, Codeberg-Infrastructure/scripted-configuration@f7177f6400 and Codeberg-Infrastructure/scripted-configuration@251b917925. It should be more targeted to full clones now, so if Cuirass is only running git fetch (or git ls-remote) and not git clone each time then it shouldn't hit the rate limiter anymore. Nothing actionable on our part to improve the situation, closing.

We deployed two improvements to the Git rate limiter, https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/commit/f7177f64002c896c82d1774a2b7740e80c49031d and https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/commit/251b9179254231c7958a5726f0a33e6dc8225b04. It should be more targeted to full clones now, so if Cuirass is only running `git fetch` (or `git ls-remote`) and not `git clone` each time then it shouldn't hit the rate limiter anymore. Nothing actionable on our part to improve the situation, closing.