Introduction

Sitemap Web Scraper (sws) is a tool for simple, flexible, and yet performant web pages scraping. It consists of a CLI that executes a Lua JIT script and outputs a CSV file.

All the logic for crawling/scraping is defined in Lua and executed on a multiple threads in Rust. The actual parsing of HTML is done in Rust. Standard CSS selectors are also implemented in Rust (using Servo's html5ever and selectors). Both functionalities are accessible through a Lua API for flexible scraping logic.

As for the crawling logic, multiple seeding options are available: robots.txt, sitemaps, or a custom HTML pages list. By default, sitemaps (either provided or extracted from robots.txt) will be crawled recursively and the discovered HTML pages will be scraped with the provided Lua script. It's also possible to dynamically add page links to the crawling queue when scraping an HTML page. See the crawl subcommand and the Lua scraper for more details.

Besides, the Lua scraping script can be used on HTML pages stored as local files, without any crawling. See the scrap subcommand doc for more details.

Furthermore, the CLI is composed of crates that can be used independently in a custom Rust program.

Getting Started

Get the binary

Download the latest standalone binary for your OS on the release page, and put it in a location available in your PATH.

Basic example

Let's create a simple urbandict.lua scraper for Urban Dictionary. Copy paste the following command:

cat << 'EOF' > urbandict_demo.lua
sws.seedPages = {
   "https://www.urbandictionary.com/define.php?term=Lua"
}

function scrapPage(page, context)
   for defIndex, def in page:select("section .definition"):enumerate() do
      local word = def:select("h1 a.word"):iter()()
      if not word then
         word = def:select("h2 a.word"):iter()()
      end
      if not word then
         goto continue
      end
      word = word:innerHtml()

      local contributor = def:select(".contributor"):iter()()
      local date = string.match(contributor:innerHtml(), ".*\\?</a>%s*(.*)\\?")
      date = sws.Date(date, "%B %d, %Y"):format("%Y-%m-%d")

      local meaning = def:select(".meaning"):iter()()
      meaning = meaning:innerText():gsub("[\n\r]+", " ")

      local example = def:select(".example"):iter()()
      example = example:innerText():gsub("[\n\r]+", " ")

      if word and date and meaning and example then
         local record = sws.Record()
         record:pushField(word)
         record:pushField(defIndex)
         record:pushField(date)
         record:pushField(meaning)
         record:pushField(example)
         context:sendRecord(record)
      end

      ::continue::
   end
end
EOF

You can then run it with:

sws crawl --script urbandict_demo.lua

As we have defined sws.seedPages to be a single page (that is Urban Dictionary's Lua definition), the scrapPage function will be run on that single page only. There are multiple seeding options which are detailed in the Lua scraper - Seed definition section.

By default the resulting csv file is written to stdout, however the -o (or --output-file) lets us specify a proper output file. Note that this file can be also be appended or truncated, using the additional flags --append or --truncate respectively. See the crawl subcommand section for me details.

Bash completion

You can source the completion script in your ~/.bashrc file with:

echo 'source <(sws completion)' >> ~/.bashrc

Subcommand: crawl

Crawl sitemaps and scrap pages content

Usage: sws crawl [OPTIONS] --script <SCRIPT>

Options:
  -s, --script <SCRIPT>
          Path to the Lua script that defines scraping logic
  -o, --output-file <OUTPUT_FILE>
          Optional file that will contain scraped data, stdout otherwise
      --append
          Append to output file
      --truncate
          Truncate output file
  -q, --quiet
          Don't output logs
  -h, --help
          Print help information

More options in CLI override

Crawler Config

The crawler configurable parameters are:

These parameters can be changed through Lua script or CLI arguments.

The priority order is: CLI (highest priority) > Lua > Default values

Lua override

You can override parameters in Lua through the global variable sws.crawlerConfig.

Here is an example of crawler configuration parmeters set using Lua:

-- You don't have to specify all parameters, only the ones you want to override.
sws.crawlerConfig = {
  userAgent = "SWSbot",
  pageBuffer = 10000,
  throttle = { Concurrent = 100 }, -- or: { PerSecond = 100 }, { Delay = 2 }
  numWorkers = 4,
  onDlError = "SkipAndLog", -- or: "Fail"
  onXmlError = "SkipAndLog",
  onScrapError = "SkipAndLog",
  robot = nil,
}

CLI override

You can override parameters through the CLI arguments.

Here is an example of crawler configuration parmeters set using CLI arguments:

sws --script path/to/scrape_logic.lua -o results.csv     \
    --user-agent     'SWSbot'                            \
    --page-buffer    10000                               \
    --conc-dl        100                                 \
    --num-workers    4                                   \
    --on-dl-error    skip-and-log                        \
    --on-xml-error   fail                                \
    --on-scrap-error skip-and-log                        \
    --robot          'https://www.google.com/robots.txt' \

Subcommand: scrap

Scrap a single remote page or multiple local pages

Usage: sws scrap [OPTIONS] --script <SCRIPT> <--url <URL>|--files <GLOB>>

Options:
  -s, --script <SCRIPT>            Path to the Lua script that defines scraping logic
      --url <URL>                  A distant html page to scrap
      --files <GLOB>               A glob pattern to select local files to scrap
  -o, --output-file <OUTPUT_FILE>  Optional file that will contain scraped data, stdout otherwise
      --append                     Append to output file
      --truncate                   Truncate output file
      --num-workers <NUM_WORKERS>  Set the number of CPU workers when scraping local files
      --on-error <ON_ERROR>        Scrap error handling strategy when scraping local files [possible values: fail, skip-and-log]
  -q, --quiet                      Don't output logs
  -h, --help                       Print help information

The parameters --url and --files are mutually exclusive (only one can be specified).

This subcommand is meant to either:

• Quickly test a Lua script on a given URL (with --url)

• Process HTML pages that have been previously stored on disk (with --files)

Lua Scraper

The scraping logic is configured through a single Lua script.

The customizable parameters are:

• seed: Defines the seed pages for crawling
• acceptUrl: A function to specify whether to accept a URL when crawling an XML Sitemap
• scrapPage: A function that defines the scraping logic for a single HTML page

Seed definition

The seed be one of seedSitemaps, seedPages, or seedRobotsTxt.

Defining a seed is always mandatory. However, when using the scrap subcommand it will be ignored as the input will be either the specified URL or the specified local files.

⚠️ Defining multiple seeds will throw an error ⚠️

Example

-- A list of sitemap URLs (gzipped sitemaps are supported)
sws.seedSitemaps = {
   "https://www.urbandictionary.com/sitemap-https.xml.gz"
}

-- A list of HTML pages
sws.seedPages = {
   "https://www.urbandictionary.com/define.php?term=Rust",
   "https://www.urbandictionary.com/define.php?term=Lua",
}

-- A single robots.txt URL
sws.seedRobotsTxt = "https://www.urbandictionary.com/robots.txt"

Robot definition

A robots.txt can be used either as:

• A crawling seed through sws.seedRobotsTxt (see above)

• A URL validation helper through sws.crawlerConfig's parameter robot (see crawler configuration)

In both cases, the resulting Robot can be used to check whether a given URL is crawlable. This Robot is available through both CrawlingContext (in acceptUrl), and ScrapingContext (in scrapPage).

The underlying Robot implementation in Rust is using the crate texting_robots.

Defining a robot is optional.

Function acceptUrl

function acceptUrl(url, context)

A Lua function to specify whether to accept a URL when crawling an XML Sitemap. Its parameters are:

• url: A URL string that is a candidate for crawling/scraping

• context: An instance of CrawlingContext

Defining acceptUrl is optional.

Example

From examples/urbandict.lua:

function acceptUrl(url, context)
   if context:sitemap() == sws.Sitemap.URL_SET then
      return string.find(url, "term=")
   else
      -- For sws.Sitemap.INDEX accept all entries
      return true
   end
end

Function scrapPage

function scrapPage(page, context)

A Lua function that defines the scraping logic for a single page. Its parameters are:

• page: The Html page being scraped

• context: An instance of ScrapingContext

Defining scrapPage is mandatory.

CSS Selectors

CSS selectors are the most powerful feature of this scraper, they are used to target and extract HTML elements in a flexible and efficient way. You can read more about CSS selectors on MDN doc, and find a good reference on W3C doc.

function scrapPage(page, context)
   for i, def in page:select("section .definition"):enumerate() do
      local word = def:select("h1 a.word"):iter()()
      print(string.format("Definition %i: %s", i, word))
   end
end

The select method is expecting a CSS selector string, its result can be either iterated or enumerated with iter and enumerate respectively. Interestingly, the elements being iterated over allow for sub selection as they also have a select method, this enables very flexible HTML elements selection.

See more details in the reference for the Select class.

Utils

Some utility functions are also exposed in Lua.

• Date utils:

  The Date helper can parse and format dates:

  local date = "March 18, 2005" -- Extracted from some page's element
  date = sws.Date(date, "%B %d, %Y"):format("%Y-%m-%d") -- Now date is "2005-03-18"
  

  Under the hood a Date wraps a Rust chrono::NaiveDate that is created using NaiveDate::parse_from_str. The format method will return a string formatted with the specified format (see specifiers for the formatting options).

Example

From examples/urbandict.lua:

function scrapPage(page, context)
   for defIndex, def in page:select("section .definition"):enumerate() do
      local word = def:select("h1 a.word"):iter()()
      if not word then
         word = def:select("h2 a.word"):iter()()
      end
      if not word then
         goto continue
      end
      word = word:innerHtml()

      local contributor = def:select(".contributor"):iter()()
      local date = string.match(contributor:innerHtml(), ".*\\?</a>%s*(.*)\\?")
      date = sws.Date(date, "%B %d, %Y"):format("%Y-%m-%d")

      local meaning = def:select(".meaning"):iter()()
      meaning = meaning:innerText():gsub("[\n\r]+", " ")

      local example = def:select(".example"):iter()()
      example = example:innerText():gsub("[\n\r]+", " ")

      if word and date and meaning and example then
         local record = sws.Record()
         record:pushField(word)
         record:pushField(defIndex)
         record:pushField(date)
         record:pushField(meaning)
         record:pushField(example)
         context:sendRecord(record)
      end

      ::continue::
   end
end

CSV Record

The Lua Record class wraps a Rust csv::StringRecord struct. In Lua it can be instantiated through sws.Record(). Its pushField(someString) method should be used to add string fields to the record.

It is possible to customize the underlying CSV Writer in Lua through the sws.csvWriterConfig table.

Example

sws.csvWriterConfig = {
   delimiter = "\t"
}

function scrapPage(page, context)
    local record = sws.Record()
    record:pushField("foo field")
    record:pushField("bar field")
    context:sendRecord(record)
end

Lua API Overview

Global variables

Namespaced variables

All the following variables are defined in the sws table.

Seeds

The configurable seed

Configurations

Types

All types are defined in the sws table.

Class Html

A parsed HTML page. Its HTML elements can be selected with CSS selectors.

Class Select

A selection made with CSS selectors. Its HTML elements can be iterated.

Class ElemRef

An HTML element reference. Its descendant HTML elements can be selected with CSS selectors.

Class Date

A helper class for parsing and formatting dates.

Class ScrapingContext

The context available when an HTML page is scraped, provided as parameter in scrapPage

Class PageLocation

The location of an HTML page.

Enum Location

Location kind.

Class Record

A dynamic CSV record. CSV formatting can be customized (see details).

Class CrawlingContext

The context available when an XML Sitemap page is crawled, provided as parameter in acceptUrl

Class Robot

Enum Sitemap

The Sitemaps formats of an XML Sitemap page.