HTTP 429 for renovate runs #2103

margau commented 2025年08月23日 13:50:42 +02:00

Comment

Hi all,
this is in response to https://social.anoxinon.de/@Codeberg/115074265437710727 (Limiting HTTP Clone).

I'm running renovate for my repositories (currently 15, but not all of them "active"), and since yesterday running into a 429 after processing the first few Repos:

"message": "Cloning into '.'...\nerror: RPC failed; HTTP 429 curl 22 The requested URL returned error: 429\nfatal: expected 'packfile'\nfatal: could not fetch 1828b7c949e7a04fe3c58ad1fdb59a459a6bc3d0 from promisor remote\nwarning: Clone succeeded, but checkout failed.\nYou can inspect what was checked out with 'git status'\nand retry with 'git restore --source=HEAD :/'\n\n",

Renovate is running inside a selfhosted forgejo runner (currently all runners are affected). I'm trying to cache, but it's currently not that easy (https://code.forgejo.org/forgejo/runner/issues/733)

Renovate currently uses an account-wide API Token to git clone using https.

Unfortunately, the renovate repo itself is currently not public yet (waiting for forgejo/forgejo#4308).

Details of the setup are also in forgejo/forgejo#8839

While I'll try to switch SSH, and don't see immediate action over here, I'd like to document the current effects here.

If possible, a more relaxed rate limit for "authenticated" clones (e.g. using api keys) would be nice of course.

Thanks!

### Comment Hi all, this is in response to https://social.anoxinon.de/@Codeberg/115074265437710727 (Limiting HTTP Clone). I'm running renovate for my repositories (currently 15, but not all of them "active"), and since yesterday running into a 429 after processing the first few Repos: ``` "message": "Cloning into '.'...\nerror: RPC failed; HTTP 429 curl 22 The requested URL returned error: 429\nfatal: expected 'packfile'\nfatal: could not fetch 1828b7c949e7a04fe3c58ad1fdb59a459a6bc3d0 from promisor remote\nwarning: Clone succeeded, but checkout failed.\nYou can inspect what was checked out with 'git status'\nand retry with 'git restore --source=HEAD :/'\n\n", ``` Renovate is running inside a selfhosted forgejo runner (currently all runners are affected). I'm trying to cache, but it's currently not that easy (https://code.forgejo.org/forgejo/runner/issues/733) Renovate currently uses an account-wide API Token to git clone using https. Unfortunately, the renovate repo itself is currently not public yet (waiting for https://codeberg.org/forgejo/forgejo/issues/4308). Details of the setup are also in https://codeberg.org/forgejo/forgejo/issues/8839 While I'll try to switch SSH, and don't see immediate action over here, I'd like to document the current effects here. If possible, a more relaxed rate limit for "authenticated" clones (e.g. using api keys) would be nice of course. Thanks!

margau commented 2025年08月23日 17:25:19 +02:00

Quick Update:
SSH Works better, but also fails consistently "mid pipeline" after a few repos (more than with HTTPS):

"message": "kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 217.197.84.140 port 22\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",

I currently see no easy workaround over here, because by design renovate does all repos it is configured for at once.

Quick Update: SSH Works better, but also fails consistently "mid pipeline" after a few repos (more than with HTTPS): ``` "message": "kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 217.197.84.140 port 22\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n", ``` I currently see no easy workaround over here, because by design renovate does all repos it is configured for at once.

fnetX commented 2025年09月04日 15:27:05 +02:00

Hi, I'm sorry to hear that our rate-limits are causing issues. We have relaxed the HTTP timeouts several times. The current limit is around 50 repos / 30 minutes (150 requests, but each clone/pull can do more than one request).

I considered fine-tuning this to separate "clones" from "pulls" so that updates to existing repos are not counted. Would this help for your situation?

For SSH, one option would be to reuse the SSH connection. Something like this in the .ssh/config

Host codeberg.org
 ControlMaster auto
 ControlPath ~/.ssh/sockets/%r@%h-%p
 ControlPersist 60

could do the trick.

Hi, I'm sorry to hear that our rate-limits are causing issues. We have relaxed the HTTP timeouts several times. The current limit is around 50 repos / 30 minutes (150 requests, but each clone/pull can do more than one request). I considered fine-tuning this to separate "clones" from "pulls" so that updates to existing repos are not counted. Would this help for your situation? For SSH, one option would be to reuse the SSH connection. Something like this in the .ssh/config ~~~ Host codeberg.org ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h-%p ControlPersist 60 ~~~ could do the trick.

rrrudolph commented 2025年09月08日 19:57:03 +02:00

Same issue here: I've tried to migrate a total of 9 repos to codeberg, but running renovate twice in succession breaks not only renovate but all workflows across my repositories (as they cannot even checkout the code anymore).

This makes it impossible to debug issues with the renovate configuration and makes workflows extremely fragile at other times.

Would it help to configure renovate with different tokens for a subset of the repos each? Or are rate limits per ip address (I only have a single server for my self-hosted runners)

Same issue here: I've tried to migrate a total of 9 repos to codeberg, but running renovate twice in succession breaks not only renovate but all workflows across my repositories (as they cannot even checkout the code anymore). This makes it impossible to debug issues with the renovate configuration and makes workflows extremely fragile at other times. Would it help to configure renovate with different tokens for a subset of the repos each? Or are rate limits per ip address (I only have a single server for my self-hosted runners)

margau commented 2025年09月09日 12:04:34 +02:00

I've added the following to my pipeline:

steps:
 - name: setup ssh config for rate limit workaround
 run: |
 mkdir -p ~/.ssh
 echo "Host codeberg.org
 ControlMaster auto
 ControlPath ~/.ssh/sockets/%r@%h-%p
 ControlPersist 60
 " > ~/.ssh/config
 mkdir -p ~/.ssh/sockets
 cat ~/.ssh/config
 - uses: https://code.forgejo.org/actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
 with:
 # persist-credentials: false
 ssh-key: "${{ secrets.SSH_KEY }}"
 ssh-known-hosts: |
 codeberg.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8hZi7K1/2E2uBX8gwPRJAHvRAob+3Sn+y2hxiEhN0buv1igjYFTgFO2qQD8vLfU/HT/P/rqvEeTvaDfY1y/vcvQ8+YuUYyTwE2UaVU5aJv89y6PEZBYycaJCPdGIfZlLMmjilh/Sk8IWSEK6dQr+g686lu5cSWrFW60ixWpHpEVB26eRWin3lKYWSQGMwwKv4LwmW3ouqqs4Z4vsqRFqXJ/eCi3yhpT+nOjljXvZKiYTpYajqUC48IHAxTWugrKe1vXWOPxVXXMQEPsaIRc2hpK+v1LmfB7GnEGvF1UAKnEZbUuiD9PBEeD5a1MZQIzcoPWCrTxipEpuXQ5Tni4mN
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7
 codeberg.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2pDxWr18SoiDJCGZ5LmxPygTlPu+cCKSkpqkvCyQzl5xmIMeKNdfdBpfbCGDPoZQghePzFZkKJNR/v9Win3Sc=
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7
 codeberg.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIVIC02vnjFyL+I4RHfvIGNtOgJMe769VTF1VR4EB3ZB
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7
 - name: setup ssh known hosts for codeberg
 run: |
 mkdir -p ~/.ssh
 echo "codeberg.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8hZi7K1/2E2uBX8gwPRJAHvRAob+3Sn+y2hxiEhN0buv1igjYFTgFO2qQD8vLfU/HT/P/rqvEeTvaDfY1y/vcvQ8+YuUYyTwE2UaVU5aJv89y6PEZBYycaJCPdGIfZlLMmjilh/Sk8IWSEK6dQr+g686lu5cSWrFW60ixWpHpEVB26eRWin3lKYWSQGMwwKv4LwmW3ouqqs4Z4vsqRFqXJ/eCi3yhpT+nOjljXvZKiYTpYajqUC48IHAxTWugrKe1vXWOPxVXXMQEPsaIRc2hpK+v1LmfB7GnEGvF1UAKnEZbUuiD9PBEeD5a1MZQIzcoPWCrTxipEpuXQ5Tni4mN
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7
 codeberg.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2pDxWr18SoiDJCGZ5LmxPygTlPu+cCKSkpqkvCyQzl5xmIMeKNdfdBpfbCGDPoZQghePzFZkKJNR/v9Win3Sc=
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7
 codeberg.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIVIC02vnjFyL+I4RHfvIGNtOgJMe769VTF1VR4EB3ZB
 # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7" >> ~/.ssh/known_hosts
 chmod 644 ~/.ssh/known_hosts
 - name: setup ssh key for codeberg
 run: |
 echo "${{ secrets.SSH_KEY }}" > ~/.ssh/id_ed25519
 chmod 600 ~/.ssh/id_ed25519
 - name: Restore renovate cache
 uses: https://code.forgejo.org/actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4
 with:
 path: |
 ${{ github.workspace }}/renovate/
 key: renovate-cache-${{ github.run_id }}
 restore-keys: |
 renovate-cache-

Renovate is configured to use SSH ("gitUrl": "ssh",) - it works a bit better, but is obviously a "not so great" workaround.

With the cache active, I think there is no "full clone" happening (didn't understood renovate there completly yet, because normally I'm nowhere near the 50 repos except maybe for checking tags of "internal" dependencies).

Of course, it would be better if there is no aggressive rate limit for authenticated/SSH sessions, but for now it is working at least twice within 10min for my amount of repos.

I've added the following to my pipeline: ``` steps: - name: setup ssh config for rate limit workaround run: | mkdir -p ~/.ssh echo "Host codeberg.org ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h-%p ControlPersist 60 " > ~/.ssh/config mkdir -p ~/.ssh/sockets cat ~/.ssh/config - uses: https://code.forgejo.org/actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5 with: # persist-credentials: false ssh-key: "${{ secrets.SSH_KEY }}" ssh-known-hosts: | codeberg.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8hZi7K1/2E2uBX8gwPRJAHvRAob+3Sn+y2hxiEhN0buv1igjYFTgFO2qQD8vLfU/HT/P/rqvEeTvaDfY1y/vcvQ8+YuUYyTwE2UaVU5aJv89y6PEZBYycaJCPdGIfZlLMmjilh/Sk8IWSEK6dQr+g686lu5cSWrFW60ixWpHpEVB26eRWin3lKYWSQGMwwKv4LwmW3ouqqs4Z4vsqRFqXJ/eCi3yhpT+nOjljXvZKiYTpYajqUC48IHAxTWugrKe1vXWOPxVXXMQEPsaIRc2hpK+v1LmfB7GnEGvF1UAKnEZbUuiD9PBEeD5a1MZQIzcoPWCrTxipEpuXQ5Tni4mN # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7 codeberg.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2pDxWr18SoiDJCGZ5LmxPygTlPu+cCKSkpqkvCyQzl5xmIMeKNdfdBpfbCGDPoZQghePzFZkKJNR/v9Win3Sc= # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7 codeberg.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIVIC02vnjFyL+I4RHfvIGNtOgJMe769VTF1VR4EB3ZB # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7 - name: setup ssh known hosts for codeberg run: | mkdir -p ~/.ssh echo "codeberg.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8hZi7K1/2E2uBX8gwPRJAHvRAob+3Sn+y2hxiEhN0buv1igjYFTgFO2qQD8vLfU/HT/P/rqvEeTvaDfY1y/vcvQ8+YuUYyTwE2UaVU5aJv89y6PEZBYycaJCPdGIfZlLMmjilh/Sk8IWSEK6dQr+g686lu5cSWrFW60ixWpHpEVB26eRWin3lKYWSQGMwwKv4LwmW3ouqqs4Z4vsqRFqXJ/eCi3yhpT+nOjljXvZKiYTpYajqUC48IHAxTWugrKe1vXWOPxVXXMQEPsaIRc2hpK+v1LmfB7GnEGvF1UAKnEZbUuiD9PBEeD5a1MZQIzcoPWCrTxipEpuXQ5Tni4mN # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7 codeberg.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2pDxWr18SoiDJCGZ5LmxPygTlPu+cCKSkpqkvCyQzl5xmIMeKNdfdBpfbCGDPoZQghePzFZkKJNR/v9Win3Sc= # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7 codeberg.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIVIC02vnjFyL+I4RHfvIGNtOgJMe769VTF1VR4EB3ZB # codeberg.org:22 SSH-2.0-OpenSSH_10.0p2 Debian-7" >> ~/.ssh/known_hosts chmod 644 ~/.ssh/known_hosts - name: setup ssh key for codeberg run: | echo "${{ secrets.SSH_KEY }}" > ~/.ssh/id_ed25519 chmod 600 ~/.ssh/id_ed25519 - name: Restore renovate cache uses: https://code.forgejo.org/actions/cache/restore@0400d5f644dc74513175e3cd8d07132dd4860809 # v4 with: path: | ${{ github.workspace }}/renovate/ key: renovate-cache-${{ github.run_id }} restore-keys: | renovate-cache- ``` Renovate is configured to use SSH (`"gitUrl": "ssh",`) - it works a bit better, but is obviously a "not so great" workaround. With the cache active, I think there is no "full clone" happening (didn't understood renovate there completly yet, because normally I'm nowhere near the 50 repos except maybe for checking tags of "internal" dependencies). Of course, it would be better if there is no aggressive rate limit for authenticated/SSH sessions, but for now it is working at least twice within 10min for my amount of repos.