Currently, there is some basic rate limiting on some HTTP backends in our HAproxy config.
There is also an open PR at Codeberg-Infrastructure/scripted-configuration#19 which shouldn't get stale and might be a good starting point, but no one in the team has yet picked up the ideas and investigate how this could be done.
Certain backends (e.g. Codeberg Pages) forward the encrypted TCP traffic into a container, so HTTP-based rate limiting is not an option.
It would be great if someone could pick up the task and help us investigate the traffic and provide patches to our rate limiting config, which reduces automated crawler traffic to a bare minimum, especially on heavy operations like generating code archives (while browsing issues etc should be mostly okay).
Currently, there is some basic rate limiting on some HTTP backends in our HAproxy config.
There is also an open PR at https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/pulls/19 which shouldn't get stale and might be a good starting point, but no one in the team has yet picked up the ideas and investigate how this could be done.
Certain backends (e.g. Codeberg Pages) forward the encrypted TCP traffic into a container, so HTTP-based rate limiting is not an option.
It would be great if someone could pick up the task and help us investigate the traffic and provide patches to our rate limiting config, which reduces automated crawler traffic to a bare minimum, especially on heavy operations like generating code archives (while browsing issues etc should be mostly okay).