Frequently Asked Questions
This is a list with frequently asked questions about web robots. Select the question to go to the answer page, or select on the eye icon after the question to show the answer in this page.
- About WWW robots
- Indexing Robots
- For Server Administrators
- Robots exclusion standard
- Why do I find entries for /robots.txt in my log files?
- How do I prevent robots scanning my site?
- Where do I find out how /robots.txt files work?
- What program should I use to create /robots.txt?
- How do I use /robots.txt on a virtual host?
- How do I use /robots.txt on a shared host?
- What about further development of /robots.txt?
- What if I cannot make a /robots.txt?
- Can I block just bad robots?
- Why did this robot ignore my /robots.txt?
- Can a /robots.txt be used in a court of law?
- Surely listing sensitive files is asking for trouble?
- About META tags
- Availability