Git hooks for large commits #418

fnetX commented 2021年03月18日 15:04:32 +01:00

Follow-up of #414 and an ongoing discussion on how to keep Codeberg clean and repo sizes low. We considered using git hooks to check for users (accidently) uploading large files in their commits that is not intended for the public. Removing this from the repo keeps it in the git history and thus it counts to the repo size (on Codeberg and on every contributors machine).

We would need a way to filter out

• accidental commits of large binary files, especially compiled software
• inclusion of build artefacts and caching (which is much harder to spot since this are small files, maybe some filtering for the names?)

There was a consideration to use git hooks for this job to check content while a user is pushing stuff to the repo. It's possible to echo messages back. Still, there must be a way to tell the server that uploading this stuff is indeed intended.

We are looking for help for the technical implementations to check the commits on receive, echo an informing message to the user and have a way to override the blocking in case the upload is intended. Any volunteers?

Note in comment

Follow-up of #414 and an ongoing discussion on how to keep Codeberg clean and repo sizes low. We considered using git hooks to check for users (accidently) uploading large files in their commits that is not intended for the public. Removing this from the repo keeps it in the git history and thus it counts to the repo size (on Codeberg and on every contributors machine). We would need a way to filter out - accidental commits of large binary files, especially compiled software - inclusion of build artefacts and caching (which is much harder to spot since this are small files, maybe some filtering for the names?) There was a consideration to use git hooks for this job to check content while a user is pushing stuff to the repo. It's possible to echo messages back. Still, there must be a way to tell the server that uploading this stuff is indeed intended. We are looking for help for the technical implementations to check the commits on receive, echo an informing message to the user and have a way to override the blocking in case the upload is intended. Any volunteers? ###### Note in comment <!-- Something like "*** WARNING *** you are about to commit a verx large file (X MB). In all common workflows this is likely unintentional and will trigger a review by the Codeberg team, unless justified for good reason possibly lead to removal or takedown. If this is intentional and required please describe reason in commit message and add the string INTENTIONAL_BIG_COMMIT in first line of your commit message. Codeberg.org is funded by volunters and contributors world-wide, please consider donating and supporting our cause!" hw in Matrix -->

hw commented 2021年03月19日 09:23:31 +01:00

There was a consideration to use git hooks ...

another option maybe cron jobs filing tickets via API into the project's issue tracker?

> There was a consideration to use git hooks ... another option maybe cron jobs filing tickets via API into the project's issue tracker?

fnetX commented 2021年03月19日 23:08:10 +01:00

Well, that is what we discussed in #414 ... but it won't prevent people from accidental uploads. Once it's in the git history, it cannot be removed without annoying everyone else.

Well, that is what we discussed in #414 ... but it won't **prevent** people from accidental uploads. Once it's in the git history, it cannot be removed without annoying everyone else.

hw commented 2021年03月21日 11:54:46 +01:00

Good point

Good point

fnetX commented 2021年05月15日 22:59:33 +02:00

Transfer from Matrix-Chat:
https://github.com/git-lfs/git-lfs/issues/282#issuecomment-296438497 links https://github.com/Ninjaccount/git-big-lfs-hook, a commit hook that might be a good starting point.

Ideally rewritten to check and inform about acceptable repo size and that git LFS is a nice thing ... but does allow the commit on second pass (or interactively or whatever ...)

Transfer from Matrix-Chat: https://github.com/git-lfs/git-lfs/issues/282#issuecomment-296438497 links https://github.com/Ninjaccount/git-big-lfs-hook, a commit hook that might be a good starting point. Ideally rewritten to *check* and *inform* about acceptable repo size and that git LFS is a nice thing ... but does allow the commit on second pass (or interactively or whatever ...)

6543 commented 2021年05月15日 23:50:16 +02:00

I would say: let them pass if commit message contain a sentence like I like to avoid lfs because ... ;)

I would say: let them pass if commit message contain a sentence like `I like to avoid lfs because ...` ;)

fnetX commented 2021年05月15日 23:54:16 +02:00

My main motivation to not strictly enforce this: What if you are collaborating on a project across instances and including the commits of others. You want to push some comment from upstream or your work from years ago to Codeberg - and ooops ... YOU SHALL NOT PASS 😂

Rewriting the history for big projects is of course an option, but not using Codeberg is probably the easier one, then.

That's why I'd go with proper information instead of enforcing 😉

My main motivation to not strictly enforce this: What if you are collaborating on a project across instances and including the commits of others. You want to push some comment from upstream or your work from years ago to Codeberg - and ooops ... ***YOU SHALL NOT PASS*** 😂 Rewriting the history for big projects is of course an option, but not using Codeberg is probably the easier one, then. That's why I'd go with proper information instead of enforcing 😉

fnetX commented 2021年11月21日 15:49:29 +01:00

PS: It might be nice to use Git push options for this job. E.g. allow users to specifiy -o disableSizeCheck.

And, have some common gitignore values to compare. I noticed quite some repos with .idea folders, and heard of many Firefox profiles committed to GitHub. It might also be sensible to react like

Hey, your commit includes a folder called ".idea" which is commonly used for IntelliJ projects and usually not necessary with your code. If you really want to include this file, please add -o disableContentCheck to your push command.

Hey, your commit includes binaries with the ending ".so", ".exe", ... which are commonly distributed as compiled results of your work and usually not necessary with your code. If you really want to include this file, please add -o disableContentCheck to your push command.

PS: It might be nice to use [Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) for this job. E.g. allow users to specifiy `-o disableSizeCheck`. And, have some common gitignore values to compare. I noticed quite some repos with `.idea` folders, and heard of many Firefox profiles committed to GitHub. It might also be sensible to react like > Hey, your commit includes a folder called ".idea" which is commonly used for IntelliJ projects and usually not necessary with your code. If you really want to include this file, please add `-o disableContentCheck` to your push command. > Hey, your commit includes binaries with the ending ".so", ".exe", ... which are commonly distributed as compiled results of your work and usually not necessary with your code. If you really want to include this file, please add `-o disableContentCheck` to your push command.

fnetX commented 2021年12月06日 03:08:14 +01:00

I ripped out the hooks on codeberg-test.org and replaced them.

find /data/git/gitea-repositories/ -type d -name hooks | xargs rm -r

Everything is now in /data/git/hooks as a Proof of Concept. I'll have a look if this is useful for Gitea, too.
Currently, in Gitea, there is one hooks folder per repo, which uses unnecessary disk space (a lot of duplication if not compressed on FS level, also with the Git templates (sample files)), and which must be updated if the Gitea installation changes for example.

Having them in one place has some benefits, I'm considering if I only change this for Codeberg (easiest) or globally (we can still check for per-repo hooks, but not use them by default). Also, this needs some testing, but I think it works fine now. If you want to confirm everything is working, feel free to heavily test Codeberg.org pushes :)

I ripped out the hooks on codeberg-test.org and replaced them. ~~~ find /data/git/gitea-repositories/ -type d -name hooks | xargs rm -r ~~~ Everything is now in `/data/git/hooks` as a Proof of Concept. I'll have a look if this is useful for Gitea, too. Currently, in Gitea, there is one hooks folder per repo, which uses unnecessary disk space (a lot of duplication if not compressed on FS level, also with the Git templates (sample files)), and which must be updated if the Gitea installation changes for example. Having them in one place has some benefits, I'm considering if I only change this for Codeberg (easiest) or globally (we can still check for per-repo hooks, but not use them by default). Also, this needs some testing, but I think it works fine now. If you want to confirm everything is working, feel free to heavily test Codeberg.org pushes :)