1
1
Fork
You've already forked scraper
0
A set of policies for organizing web scraping code
  • Common Lisp 100%
Find a file
2026年05月15日 07:49:18 -07:00
src Add a new load-resource method for lists 2026年05月15日 05:11:46 -07:00
.gitignore Initial commit 2026年03月31日 07:51:11 -07:00
LICENSE Add MIT license 2026年04月09日 11:42:36 -07:00
README.org Highlight 2026年05月15日 07:49:18 -07:00
scraper.asd Bump version to 1.0.1 2026年04月09日 11:43:12 -07:00

scraper

About

  • This is a small library that provides some policy on how to organize web scraping code.
  • It wants to know the following.

  • WHAT kind of page do you want to process?

    • This is accomplished by making small subclasses of abstract-page.

  • WHERE is the page?

    • This is handled polymorphically by load-resource and load-resource-as-dom.
(process (page 'all-images "https://metacpan.org/"))
;; WHAT WHERE

That's all this does. It just provides a little bit of structure.

I really hesitated to call this a library, because it's so small. I think of it more as a policy for you to follow.

Extending Scraper For Your Own Use

Package Setup

  • This is how I would pull in scraper into your own package.
(defpackage whatever
 (:use ; Pull in the libraries you want to use.
 :cl ; This is a very minimal example.
 :quri) ; Feel free to pull in more libraries that you like.
 (:import-from #:lquery
 #:$
 #:1ドル)
 (:import-from #:scraper ; Import these symbols from scraper.
 #:abstract-page
 #:src
 #:process
 #:load-resource-as-dom
 #:page))
(in-package :whatever)

Create a subclass of abstract-page. 

(defclass all-images (abstract-page)
 ()
 (:documentation "This is a general page type for scraping all images from a page."))
  • The main purpose of creating a subclass is to allow process to dispatch on this new type.
  • Subclasses of abstract-page inherit a slot named src.
  • If you want to add more slots for your own use, you may.

Implement a process method that dispatches on your new type. 

(defmethod process ((page all-images))
 "Return a list of all images from the `page'."
 (let* ((dom (load-resource-as-dom (src page)))
 (images ($ dom "img" (attr "src"))))
 (coerce images 'list)))
  • A process method should take a page as a parameter.
  • However, there are no restrictions on what it returns as long as it makes sense to you.

Example Usage

Basic Usage

First, let's look at the most typical way this would be used.

;; For demonstration purposes, go into the scraper package
;; to make symbols easily accessible.
(ql:quickload :scraper)
(in-package :scraper)
(defparameter mi (page 'all-images "https://metacpan.org/"))
(process mi)
("/static/images/metacpan-logo.svg" "/static/images/metacpan-logo.svg"
 "/static/images/sponsors/bytemark_logo.svg"
 "/static/images/sponsors/datadog.svg" "/static/images/sponsors/deriv.svg"
 "/static/images/sponsors/elastic.svg"
 "/static/images/sponsors/fastly_logo.svg"
 "/static/images/sponsors/geocodelogo.svg"
 "/static/images/sponsors/iban-logo.png"
 "/static/images/sponsors/liquidweb_logo.png"
 "/static/images/sponsors/open-cage.svg"
 "/static/images/sponsors/perl-onion.svg")

Polymorphism on Resource Type

What if the HTML you want is in the file system?

cd /tmp
wget https://metacpan.org/
# saved as /tmp/index.html

We can use the same process method on a different source.

(defparameter mi-local (page 'all-images "file:///tmp/index.html"))
(process mi-local)
("/static/images/metacpan-logo.svg" "/static/images/metacpan-logo.svg"
 "/static/images/sponsors/bytemark_logo.svg"
 "/static/images/sponsors/datadog.svg" "/static/images/sponsors/deriv.svg"
 "/static/images/sponsors/elastic.svg"
 "/static/images/sponsors/fastly_logo.svg"
 "/static/images/sponsors/geocodelogo.svg"
 "/static/images/sponsors/iban-logo.png"
 "/static/images/sponsors/liquidweb_logo.png"
 "/static/images/sponsors/open-cage.svg"
 "/static/images/sponsors/perl-onion.svg")

You can also work directly with plump-dom elements.

(defparameter dom (load-resource-as-dom (uri "file:///tmp/index.html")))
(process (page 'all-images dom))
("/static/images/metacpan-logo.svg" "/static/images/metacpan-logo.svg"
 "/static/images/sponsors/bytemark_logo.svg"
 "/static/images/sponsors/datadog.svg" "/static/images/sponsors/deriv.svg"
 "/static/images/sponsors/elastic.svg"
 "/static/images/sponsors/fastly_logo.svg"
 "/static/images/sponsors/geocodelogo.svg"
 "/static/images/sponsors/iban-logo.png"
 "/static/images/sponsors/liquidweb_logo.png"
 "/static/images/sponsors/open-cage.svg"
 "/static/images/sponsors/perl-onion.svg")

This lets you even use the same scraping code on a specific part of the document.

(process (page 'all-images ($ dom ".footer-social")))
("/static/images/metacpan-logo.svg")

Polymorphism on Page Type

What if I wanted to parse feeds instead of images while using the same source?

(defparameter mf (page 'all-feeds "https://metacpan.org/"))
(process mf)
((:HREF "/recent.rss" :TYPE "application/rss+xml" :TITLE
 "Recent CPAN Uploads - MetaCPAN"))

Changing the type from all-images to all-feeds changes what process method gets used. Notice that the return value has a different shape that the previous calls the process. Specializations of process are allowed to return data in any shape that makes sense to the author.

API Reference

Class: abstract-page

  • Whenever you want to scrape a certain kind of page, you make a subclass of abstract-page.
  • It must contain a src slot that lets load-resource return the data of the resource.

  • A few concrete page subclasses have been provided as examples.

    • all-images
    • all-feeds

Generic Function: (load-resource src)

  • This is a generic function that dispatches on various types to return the data of a resource.
  • It's usually being passed the value of the src slot of a page instance.
  • Whether a page is being loaded from the web or frome the file system, load-resource should be able to handle it.
Supported Types Description
quri:uri-http Load from the web. Both HTTP and HTTPS are supported.
quri:uri-file Load from the filesystem.
simple-string Use the given string as the source.
cons Assumed to be a list of strings that can be concatenated to form one document
vector Assumed to be a vector of plump-dom elements
  • This should be enough for most people, but you could always add more if you have another data source you want to support.

Function: (load-resource-as-dom src)

  • This function uses load-resource and converts the result to a vector of plump-dom elements that are ready to be queried.
  • This is the function that should be used when implementing your own process methods.
;; load via HTTP
(load-resource-as-dom (uri "https://codeberg.org/"))
;; load via filesystem
(load-resource-as-dom (uri "file:///tmp/index.html"))

Function: (page subtype src)

  • This is a convenience function for instantiating page subtypes.
  • It saves a little bit of typing.
(page 'all-images "https://slashdot.org/")
;; is the same as
(make-instance 'all-images :src (uri "https://slashdot.org/"))
  • To pass in a string that you don't want to be treated as a URI, wrap it in a list.
  • This can be useful to those who want to fetch HTML in another way.
(page 'all-images (list "<html></html>"))

Generic Function: (process page)

  • Every page type should have its own specialized process method for doing the actual scraping.
(defparameter si (page 'all-images "https://slashdot.org/"))
(defparameter images (process si))

Variable: *proxy*

  • As a convenience, an HTTP proxy may be specified here if you need one.
  • Anything that dexador can use as a proxy is valid here.
(setq *proxy* "http://user:password@myproxy:3128")

Variable: *headers*

  • As a convenience, HTTP headers can be set here if needed.
  • Anything that dexador can use as a header is valid here.
(let* ((*headers* '(("User-Agent" . "FooBar")))
 (res (load-resource (uri "https://httpbin.org/anything?q=life"))))
 (princ res))
{
 "args": {
 "q": "life"
 },
 "data": "",
 "files": {},
 "form": {},
 "headers": {
 "Accept": "*/*",
 "Content-Length": "0",
 "Host": "httpbin.org",
 "User-Agent": "FooBar"
 },
 "json": null,
 "method": "GET",
 "url": "https://httpbin.org/anything?q=life"
}

Resources

See Also