Query Web Archive Crawl Indexes (‘CDX’)
Methods are provided to retrieve web archive crawl index (‘CDX’) metadata and directly query the ‘CDX’ ‘API’ endpoint to retrieve mementos for a given set of parameters.
The following functions are implemented:
cdx_query: Query a CDX index endpointfetch_collections_index: Fetch collections index
devtools::install_github("hrbrmstr/cdx")
library(cdx) library(tidyverse) # current verison packageVersion("cdx")
## [1] '0.1.0'
cidx <- fetch_collections_index() rprj <- cdx_query(cidx$cdx_api[1], "*.r-project.org") rprj
## # A tibble: 14,358 x 12
## urlkey timestamp length url mime_detected offset mime filename status languages charset digest
## * <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 org,r-project)/ 20180818010708 2835 http:... text/html 567295... text... crawl-data... 200 eng UTF-8 WTRYJ...
## 2 org,r-project)/ 20180819003057 700 http:... text/html 7358684 text... crawl-data... 301 <NA> <NA> RGN4E...
## 3 org,r-project)/ 20180819003057 2834 http:... text/html 550642... text... crawl-data... 200 eng UTF-8 WTRYJ...
## 4 org,r-project)/ 20180819003103 702 https... text/html 207506... text... crawl-data... 301 <NA> <NA> AMYSZ...
## 5 org,r-project)/ 20180819003103 2839 https... text/html 971885... text... crawl-data... 200 eng UTF-8 WTRYJ...
## 6 org,r-project)/ 20180819194145 2832 http:... text/html 537747... text... crawl-data... 200 eng UTF-8 WTRYJ...
## 7 org,r-project)/ 20180820013726 2832 http:... text/html 570313... text... crawl-data... 200 eng UTF-8 WTRYJ...
## 8 org,r-project)/ 20180820102922 556 https... <NA> 256576... warc... crawl-data... 304 <NA> <NA> 3I42H...
## 9 org,r-project)/ 20180821155855 702 https... text/html 208822... text... crawl-data... 301 <NA> <NA> AMYSZ...
## 10 org,r-project)/ 20180821155857 2837 https... text/html 962600... text... crawl-data... 200 eng UTF-8 WTRYJ...
## # ... with 14,348 more rows