LFS easily hits requests limits to hit 429 codes #2242

bobbens commented 2025年11月30日 12:43:02 +01:00

Comment

We've been trying to reduce bandwidth usage at naev by switching to LFS. The results are great in terms of saving bandwidth and disk space, but there's a small issue that the assets repo has 2000+ small files in LFS (total <200 MiB). Apparently, these files count towards the request limit which I believe is 4,000 requests per 30 minutes. So just doing a checkout + a push would trigger the temporary 429 status. This is not something we ran into issues with when we didn't use LFS, despite the repo being almost 3x larger.

We've been able to workaround this issue with continuous integration by using heavy caching, however, I still hit this personally while working on the repository. My question is, would it be possible to split the LFS usage from normal browsing for the request limit or have some other way to mitigate the issue when using repos with lots of small files? An extreme example would be a repo with 4,000 small LFS files which would not allow cloning in one go, which is not an issue with a normal git repo no matter how large it is.

### Comment We've been trying to reduce bandwidth usage at [naev](https://codeberg.org/naev/naev) by switching to LFS. The results are great in terms of saving bandwidth and disk space, but there's a small issue that the assets repo has 2000+ small files in LFS (total <200 MiB). Apparently, these files count towards the request limit which I believe is 4,000 requests per 30 minutes. So just doing a checkout + a push would trigger the temporary 429 status. This is not something we ran into issues with when we didn't use LFS, despite the repo being almost 3x larger. We've been able to workaround this issue with continuous integration by using heavy caching, however, I still hit this personally while working on the repository. My question is, **would it be possible to split the LFS usage from normal browsing for the request limit** or have some other way to mitigate the issue when using repos with lots of small files? An extreme example would be a repo with 4,000 small LFS files which would not allow cloning in one go, which is not an issue with a normal git repo no matter how large it is.

fnetX commented 2025年12月03日 12:05:59 +01:00

Thank you for the feedback, I acknowledge this being a problem. I assume the limit mostly happens on initial clones, and updates of large repos only fetches a few new objects?

I haven't looked in how pushing LFS files work. Does it do a request for every file to see if it already exists on the server?

Thank you for the feedback, I acknowledge this being a problem. I assume the limit mostly happens on initial clones, and updates of large repos only fetches a few new objects? I haven't looked in how pushing LFS files work. Does it do a request for every file to see if it already exists on the server?

bobbens commented 2025年12月03日 12:29:53 +01:00

@fnetX wrote in #2242 (comment):

Thank you for the feedback, I acknowledge this being a problem. I assume the limit mostly happens on initial clones, and updates of large repos only fetches a few new objects?

Yes, it was quite a problem with our CI actions, but we were able to solve it with lots of caching (which is better for the long run I guess).

I haven't looked in how pushing LFS files work. Does it do a request for every file to see if it already exists on the server?

Sorry for the confusion, by push, I meant changing changing many small files. I do not believe pushing without changing files would trigger it.

I do not think it does a request for every file, only when it sees a pointer to a file that is not locally cached.

@fnetX wrote in https://codeberg.org/Codeberg/Community/issues/2242#issuecomment-8663982: > Thank you for the feedback, I acknowledge this being a problem. I assume the limit mostly happens on initial clones, and updates of large repos only fetches a few new objects? Yes, it was quite a problem with our CI actions, but we were able to solve it with lots of caching (which is better for the long run I guess). > I haven't looked in how pushing LFS files work. Does it do a request for every file to see if it already exists on the server? Sorry for the confusion, by push, I meant changing changing many small files. I do not believe pushing without changing files would trigger it. I do not think it does a request for every file, only when it sees a pointer to a file that is not locally cached.

fnetX commented 2025年12月03日 12:38:24 +01:00

I have excluded LFS from rate-limiting for now. But indeed, caching them locally is for us much better than downloading 2GB of assets for every CI run.

I have excluded LFS from rate-limiting for now. But indeed, caching them locally is for us much better than downloading 2GB of assets for every CI run.