A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker [R package robotstxt version 0.7.15]

robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Version: 0.7.15

Depends: R (≥ 3.0.0)

Imports: stringr (≥ 1.0.0), httr (≥ 1.0.0), spiderbar (≥ 0.2.0), future.apply (≥ 1.0.0), magrittr, utils

Suggests: knitr, rmarkdown, dplyr, testthat, covr, curl

Published: 2024年08月29日

DOI: 10.32614/CRAN.package.robotstxt

Author: Pedro Baltazar [aut, cre], Peter Meissner [aut], Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.), Oliver Keys [ctb] (original release code review), Rich Fitz John [ctb] (original release code review)

Maintainer: Pedro Baltazar <pedrobtz at gmail.com>

BugReports: https://github.com/ropensci/robotstxt/issues

License: MIT + file LICENSE

URL: https://docs.ropensci.org/robotstxt/, https://github.com/ropensci/robotstxt

NeedsCompilation: no

Materials: NEWS

In views: WebTechnologies

CRAN checks: robotstxt results