Keeping AI companies from copyright-violating a website for LLM training is difficult, but not impossible. I got pretty far using Apache .htaccess.
|
|
||
|---|---|---|
| identified | ||
| masquerade | Upload files to "/" | |
| README | ||
This project is to stop (LLM training) crawlers from copyright-thefting my website. I find these bots with a hidden-link tarpit or by looking for single access events (no css), which I then ban if they come from a cloud server. So far I have learned: • Amazon AWS, Google Cloud, Microsoft Azure, and Chinese telecom companies are pretty easy to block. These were the early heavy hitters. • Huawei has little cloud server farms all over the world. • Some mysterious entity rents servers all over the world and crawls by sniping one resource at a time. The snipes come in clusters, so all these bots are running the same crawler software, with some but not complete inter-communication. Popular cloud companies are OVH Cloud, EGI Hosting, Web2Objects, Host Royale, Digital Ocean, Cloud Innovation, .... Files are Apache .htaccess snippets to keep out AI crawlers. The 'identified' file assumes that the robots.txt file matches https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt Skewray Research, LLC https://www.skewray.com Public Domain <http://unlicense.org>