I have a Magento installation with multiple stores, and I need to have different robots.txt files for each store. Currently, I have a single robots.txt file in the root directory of my Magento installation, but I want to customize it separately for each store.
Is there a specific configuration or code modification that needs to be done to have different robots.txt files for each store? I want to ensure that each store has its own set of rules and directives within the robots.txt file.
Any guidance or code examples would be greatly appreciated. Thank you!
2 Answers 2
By default, you shouldn't create robots.txt manually in the webroot. Just update the Store configuration in: Content > Design > Configuration > Edit Custom instruction of robots.txt for each stores.
At some point Magento added a module called Magento_Robots that will dynamically render robots.txt content for the website being viewed.
Everything is already wired up, so all you would need to do is:
- Set the robots.txt for each website in admin from: Content > Design > Configuration > Search Engine Robots > Edit custom instruction of robots.txt file
- Make sure whether your various stores are loading from separate domains or loading from subfolders....that the correct store scope / store code is used to run the app
Note: Looking at the robots model, it looks like it only fetches robots.txt configuration data from as low as the website scope. I think that makes sense sense store views are typically used for languages
Robots Router
The \Magento\Robots\Controller\Router::match() router matches requests to /robots.txt and dispatches a robots controller action
Robots Action
The \Magento\Robots\Controller\Index\Index::execute() action adds robots_index_index layout handle and sets the result pages content type to plain text.
Robots Layout + Block + Model
The \Magento\Robots\Block\Data::_toHtml() outputs robots.txt content from the \Magento\Robots\Model\Robots::getData() method. The data for robots.txt is fetched from the design/search_engine_robots/custom_instructions system configuration.
