[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday 05 May 2004 15:59, George Theall wrote: | I maintain a few small sites, monitor my logs pretty closely, and have a | couple of traps for bad robots, including a bogus setting in my | robots.txt files telling robots not to visit a non-existent area of my | webs. While I do find plenty of examples of 'bots that completely | ignore restrictions in robots.txt, I can't recall the last time I saw Specifically, Yahoo!'s Slurp and MSN's crawler _WILL_ ignore robots.txt sometimes, but its not clear when (not clear to me, anyway). - -- Tobias DiPasquale 202A 04C4 2CE6 B985 8520 88D6 CD25 1A6C B9B5 1595 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFAmVAHzSUabLm1FZURAqRvAKCYNJDl9losswRk0VmazIMDhH13agCfa3b3 pYuozaErDXBVIwgf6Qc9fPI= =9TyK -----END PGP SIGNATURE----- ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug