The robots.txt appears to potentially be incorrectly formatted #2856

ell1e commented 2026年07月10日 12:41:27 +02:00

Comment

The robots.txt appears to potentially be incorrectly formatted. Currently, one section looks like this:

User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Diffbot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: meta-externalagent
User-agent: OAI-SearchBot
User-agent: Omgili
User-agent: Omgilibot
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: Scrapy
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: YouBot
Disallow: /

I don't think grouping user agents like that is possible. I'm pretty sure this just means all bots other than YouBot will ignore this section.

Therefore, I think for this to be effective, it would need to be formatted like this:

User-agent: Amazonbot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: facebookexternalhit
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ICC-Crawler
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: img2dataset
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: VelenPublicWebCrawler
Disallow: /
User-agent: YouBot
Disallow: /

### Comment The [robots.txt](https://codeberg.org/robots.txt) appears to potentially be incorrectly formatted. Currently, one section looks like this: ``` User-agent: Amazonbot User-agent: anthropic-ai User-agent: Applebot-Extended User-agent: Bytespider User-agent: CCBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-Web User-agent: cohere-ai User-agent: Diffbot User-agent: FacebookBot User-agent: facebookexternalhit User-agent: FriendlyCrawler User-agent: Google-Extended User-agent: GPTBot User-agent: ICC-Crawler User-agent: ImagesiftBot User-agent: img2dataset User-agent: meta-externalagent User-agent: OAI-SearchBot User-agent: Omgili User-agent: Omgilibot User-agent: PerplexityBot User-agent: PetalBot User-agent: Scrapy User-agent: Timpibot User-agent: VelenPublicWebCrawler User-agent: YouBot Disallow: / ``` I don't think grouping user agents like that is possible. I'm pretty sure this just means all bots other than `YouBot` will ignore this section. Therefore, I think for this to be effective, it would need to be formatted like this: ``` User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: img2dataset Disallow: / User-agent: meta-externalagent Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Omgili Disallow: / User-agent: Omgilibot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Scrapy Disallow: / User-agent: Timpibot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: YouBot Disallow: / ```

zoe commented 2026年07月10日 13:29:58 +02:00

Hi there,

the existing robots.txt is correct imo.
RFC 9309 allows grouping of user-agents aswell as disallow paths.
Googles Crawling Infrastructure even shows a grouped example here https://developers.google.com/crawling/docs/robots-txt/create-robots-txt?hl=de#create_rules

Hi there, the existing robots.txt is correct imo. RFC 9309 allows grouping of user-agents aswell as disallow paths. Googles Crawling Infrastructure even shows a grouped example here https://developers.google.com/crawling/docs/robots-txt/create-robots-txt?hl=de#create_rules

ell1e commented 2026年07月10日 13:57:04 +02:00

I see, interesting! I still wouldn't risk it, because some people may just use the original site which details the original method by Martijn Koster in 1994, while RFC 9309 is an extension not all bots may know. The original site offers no such grouping functionality: https://www.robotstxt.org/robotstxt.html But it's nice to hear that at least Google will implement this.

I see, interesting! I still wouldn't risk it, because some people may just use the original site which details the original method by Martijn Koster in 1994, while RFC 9309 is an extension not all bots may know. The original site offers no such grouping functionality: https://www.robotstxt.org/robotstxt.html But it's nice to hear that at least Google will implement this.

ell1e commented 2026年07月10日 14:08:50 +02:00

Update: somebody in chat pointed out that https://www.robotstxt.org/orig.html has this section:

If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.

So even the old standard has this and I was just wrong. My apologies!

Update: somebody in chat pointed out that https://www.robotstxt.org/orig.html has this section: > If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record. So even the old standard has this and I was just wrong. My apologies!