What can I do to restrict commercial AI/LLM (OpenAI/ChatGPT) training as much as possible? #1585

nej commented 2024年06月10日 07:16:26 +02:00

Comment

Hello,
I love the idea of developing and publishing FOSS, especially based on AGPL licenses. But I don't want my code to be used for AI training of those big commercial LLMs like ChatGPT from OpenAI. Using Codeberg for sure is one big step forward as opposed to publishing code on GitHub. Is there anything specifically for Codeberg that I can do to make AI training from published code here as hard as possible?

Some thoughts:

• License-wise, According to https://docs.codeberg.org/getting-started/licensing/#license-decision-diagram AGPL-3.0 seems like a good choice. But having search for licenses that forbid AI training, there still seems to be a legal gray zone as of today.
• Private repositories would be kinda against the whole FOSS topic. I really would like to make code available for non-commercial use. I just don't want to foster big tech and make it even bigger.
• More specifically: what might prevent big tech from just grabbing all code here by web scraping? Might password-protected access for repositories or some kind of invite-code be one solution to at least raise the barrier for automated scraping a bit?
• Blacklisting of large LLM ASNs/IP ranges?
• CAPTCHAs? Although I really find them annoying. Also I am not sure, if AI can solve those better than humans anyway.
• Other alternatives for scraping detection
• I think this topic would make a great FAQ entry as well

Kind regards
nej

### Comment Hello, I love the idea of developing and publishing FOSS, especially based on AGPL licenses. But I don't want my code to be used for AI training of those big commercial LLMs like ChatGPT from OpenAI. Using Codeberg for sure is one big step forward as opposed to publishing code on GitHub. Is there anything specifically for Codeberg that I can do to make AI training from published code here as hard as possible? Some thoughts: - License-wise, According to https://docs.codeberg.org/getting-started/licensing/#license-decision-diagram AGPL-3.0 seems like a good choice. But having search for licenses that forbid AI training, there still seems to be a legal gray zone as of today. - Private repositories would be kinda against the whole FOSS topic. I really would like to make code available for non-commercial use. I just don't want to foster big tech and make it even bigger. - More specifically: what might prevent big tech from just grabbing all code here by web scraping? Might password-protected access for repositories or some kind of invite-code be one solution to at least raise the barrier for automated scraping a bit? - Blacklisting of large LLM ASNs/IP ranges? - CAPTCHAs? Although I really find them annoying. Also I am not sure, if AI can solve those better than humans anyway. - Other alternatives for scraping detection - I think this topic would make a great FAQ entry as well Kind regards nej

fnetX commented 2024年06月11日 13:46:40 +02:00

The problem is: If you make it slightly harder for big corps to scrape codeberg, you likely make it much harder for human beings to use your code. You can set your user visibility to "Limited", which means only people with an account can interact with your repos. But it makes cloning from the cli impossible without signing in, for example.

There is not much you can do. Codeberg actively opted out of some crawlers via our robots.txt file, and we are regularly blocking IP ranges which do not respect our robots.txt (some AI startups listing only their key inevstors instead of explaining what they actually do, SEO optimization services, lots of stuff from China).

It's hard to prevent this, after all. There is also Software Heritage which partnered with AI and also has mirrored Codeberg content (mostly for a good mission, but this news was a bummer).

It's really hard to escape, but we're trying to limit reuse for AI as much as possible (also in part due to us being annoyed by most traffic to Codeberg originating from bots instead of humans).

The problem is: If you make it slightly harder for big corps to scrape codeberg, you likely make it much harder for human beings to use your code. You can set your user visibility to "Limited", which means only people with an account can interact with your repos. But it makes cloning from the cli impossible without signing in, for example. There is not much *you* can do. Codeberg actively opted out of some crawlers via our [robots.txt file](https://codeberg.org/robots.txt), and we are regularly blocking IP ranges which do not respect our robots.txt (some AI startups listing only their key inevstors instead of explaining what they actually do, SEO optimization services, lots of stuff from China). It's hard to prevent this, after all. There is also [Software Heritage which partnered with AI](https://www.softwareheritage.org/2024/02/28/responsible-ai-with-starcoder2/) and also has mirrored Codeberg content (mostly for a good mission, but this news was a bummer). It's really hard to escape, but we're trying to limit reuse for AI as much as possible (also in part due to us being annoyed by most traffic to Codeberg originating from bots instead of humans).

nej commented 2024年06月11日 17:45:00 +02:00

Thanks very much for your thorough answer @fnetX .
It is good to know that you already try to actively block some of the major evils. Also understandable is that user experience should not suffer too much from anti-scraping mechanisms. Personally I think, most privacy-aware users/developers wouldn't mind trading some convenience to protect their code. At least for me that is one of the main arguments to appreciate Codeberg.

As I just have discovered this platform, I am not aware of its features yet. Your suggestion of "Limited" user visibility sounds like a first good step - I'd consider the need for an account to access repos as a low barrier. This optimally should exclude all those anonymous scrapers. Whereas having a "Private" repo and need to invite collaborators would be much more cumbersome and probably not feasible with increasing amount of users.

Hmm and the softwareheritage.org topic seems like the exact thing I'd like to avoid. I suppose, even with limited visibility, an organization with such intentions isn't stopped by need to make an account - manually or bot.

// update
Just heared of that organization, and hence I might be mislead: But just seeing the name Nvidia already makes me sceptical and hints, where this might be going.

Thanks very much for your thorough answer @fnetX . It is good to know that you already try to actively block some of the major evils. Also understandable is that user experience should not suffer too much from anti-scraping mechanisms. Personally I think, most privacy-aware users/developers wouldn't mind trading some convenience to protect their code. At least for me that is one of the main arguments to appreciate Codeberg. As I just have discovered this platform, I am not aware of its features yet. Your suggestion of "Limited" user visibility sounds like a first good step - I'd consider the need for an account to access repos as a low barrier. This optimally should exclude all those anonymous scrapers. Whereas having a "Private" repo and need to invite collaborators would be much more cumbersome and probably not feasible with increasing amount of users. Hmm and the softwareheritage.org topic seems like the exact thing I'd like to avoid. I suppose, even with limited visibility, an organization with such intentions isn't stopped by need to make an account - manually or bot. // update Just heared of that organization, and hence I might be mislead: But just seeing the name Nvidia already makes me sceptical and hints, where this might be going.

cfr commented 2024年08月19日 08:54:35 +02:00

If you only want to allow non-commercial use, you don't want a FOSS licence. Any licence which meets standard(ish?) criteria for free software will necessarily allow commercial use.

But there's a difference between limitations on access to software and limitations on the use of software you provide access to.

If you only want to allow non-commercial use, you don't want a FOSS licence. Any licence which meets standard(ish?) criteria for free software will necessarily allow commercial use. But there's a difference between limitations on access to software and limitations on the use of software you provide access to.

n0toose commented 2024年08月19日 18:44:26 +02:00

If you only want to allow non-commercial use, you don't want a FOSS licence. Any licence which meets standard(ish?) criteria for free software will necessarily allow commercial use.

This is not really true, commercial use is allowed under specific terms and conditions, such as, for example, crediting the original author in (commercial) composites and derivatives of your work.

> If you only want to allow non-commercial use, you don't want a FOSS licence. Any licence which meets standard(ish?) criteria for free software will necessarily allow commercial use. This is not really true, commercial use is allowed *under specific terms and conditions*, such as, for example, crediting the original author in (commercial) composites and derivatives of your work.

n0toose commented 2024年08月19日 18:45:00 +02:00

The original thread has served its purpose, I believe, therefore I'm closing this...

Feel free to ping me in order to have it reopened.

The original thread has served its purpose, I believe, therefore I'm closing this... Feel free to ping me in order to have it reopened.

strogonoff commented 2025年05月24日 11:41:38 +02:00

Among other hosters, Sourcehut allegedly implements measures aimed to curtail scraping.

Among other hosters, Sourcehut allegedly implements measures aimed to curtail scraping.