Module robotparser
[]
Module robotparser
robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package: 1) GNU
GPLv2 2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
RobotFileParser
This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
|
RuleLine
A rule line is a single "Allow:" (allowance==True) or
"Disallow:" (allowance==False) followed by a path.
|
Entry
An entry has one or more user-agents and zero or more rulelines
|
|
URLopener
|
|
|
_debug(msg)
|
|
|
_check(a,
b)
|
|
|
_test()
|