Self-hosted actions runners get rate limited #1698

linus commented 2024年11月09日 20:16:25 +01:00

Comment

We (@sdomi and myself) host two, soon three, separate forgejo actions runners connected to Codeberg. As far as I can tell this is necessary because there can only be one registration of a runner to an organization or user, and they are for https://codeberg.org/linusg, https://codeberg.org/kiesel-js, and https://codeberg.org/servfail respectively.

Because they all run on the same host we sometimes get rate limited. I'm not sure if the forgejo actions runner is expected to handle this gracefully, perhaps it would after #944? At the moment it seems it just keeps sending out more requests and the only solution is to manually stop the affected runner(s), wait for some time, and restart it.

We tried bumping the fetch interval from the default of 2s to 20s and still ran into issues, for now I set them all to 60s which is not acceptable IMO.

Would it be possible to do either of these until the runner handles this in a better way?

• Increase the rate limit for runner-specific endpoints
• Increase the rate limit for specific users/orgs (I'm a member of Codeberg e.V. and actively use the platform for all of my current projects)
• Increase the rate limit for this specific host (185.236.240.103 / 2a0d:eb00:8006:acab:212:fdff:feaa:aaaa, aka sakamoto.pl)

Thanks!

### Comment We (@sdomi and myself) host two, soon three, separate forgejo actions runners connected to Codeberg. As far as I can tell this is necessary because there can only be one registration of a runner to an organization or user, and they are for https://codeberg.org/linusg, https://codeberg.org/kiesel-js, and https://codeberg.org/servfail respectively. Because they all run on the same host we sometimes get rate limited. I'm not sure if the forgejo actions runner is expected to handle this gracefully, perhaps it would after #944? At the moment it seems it just keeps sending out more requests and the only solution is to manually stop the affected runner(s), wait for some time, and restart it. We tried bumping the fetch interval from the default of 2s to 20s and still ran into issues, for now I set them all to 60s which is not acceptable IMO. Would it be possible to do either of these until the runner handles this in a better way? - Increase the rate limit for runner-specific endpoints - Increase the rate limit for specific users/orgs (I'm a member of Codeberg e.V. and actively use the platform for all of my current projects) - Increase the rate limit for this specific host (`185.236.240.103` / `2a0d:eb00:8006:acab:212:fdff:feaa:aaaa`, aka `sakamoto.pl`) Thanks!

sdomi commented 2024年11月09日 21:50:14 +01:00

sidenote: 2s is the default in the example config file; in our case, it was actually set to 4s when we first investigated the issue today. it would be nice if forgejo did long polling on this...

sidenote: 2s is the default in the example config file; in our case, it was actually set to 4s when we first investigated the issue today. it would be nice if forgejo did long polling on this...

fnetX commented 2024年11月09日 22:50:23 +01:00

AFAIK the Forgejo runner automatically sets the value to at least 30 seconds when it connects to codeberg.

I'll add an exception for you, though.

Improving rate-limiting w.r.t. Forgejo Actions (also client-side when viewing logs!) is a high priority, but it will take some time.

AFAIK the Forgejo runner automatically sets the value to at least 30 seconds when it connects to codeberg. I'll add an exception for you, though. Improving rate-limiting w.r.t. Forgejo Actions (also client-side when viewing logs!) is a high priority, but it will take some time.

fnetX commented 2024年11月10日 03:02:37 +01:00

The IP addresses have been allowlisted explicitly.

The IP addresses have been allowlisted explicitly.

linus commented 2024年11月10日 03:07:49 +01:00

Thank you!

Thank you!