Command line

install the megalania package for the megalania command. Set MEG_API_KEY, then megalania scrape https://docs.megalania.io/. The key is read only from MEG_API_KEY, never from a flag, so it does not end up in your shell history.

Install

Python 3.10 or newer. pipx keeps it apart from your projects:

pipx install megalania

One page

The body goes to stdout. The credits and the request id go to stderr.

megalania scrape https://docs.megalania.io/
megalania scrape https://docs.megalania.io/ --response-types text --country-code DE -o page.txt

Flags: --response-types, --engine, --proxy-tier, --country-code and --max-credits. Any other key of the JSON envelope goes in --json, as a JSON object, or - to read it from stdin:

megalania scrape https://docs.megalania.io/ --json '{"only_main_content": false}'

Before you try a site

Free: what one page of the site costs on each route (tier).

megalania tier docs.megalania.io

A site’s URLs

megalania map https://docs.megalania.io/ --json '{"search": "guides", "limit": 50}'

Batches and crawls

batch create prints the batch. batch wait waits until it is done, failed or canceled. batch results downloads every finished page into a folder, one file per item, named by the item id.

id=$(megalania batch create https://docs.megalania.io/ https://docs.megalania.io/guides/fetch | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])')
megalania batch wait "$id" --timeout 600
megalania batch results "$id" --dir pages

A crawl starts from one URL:

megalania batch create https://docs.megalania.io/ --crawl '{"max_pages": 50, "max_depth": 2}'

--file urls.txt reads one URL per line. batch get ID and batch cancel ID call the batch API.

Monitors

megalania monitor list
megalania monitor get <monitor id>
megalania monitor checks <monitor id>

monitor create --json, update ID --json, run ID and delete ID take the bodies of the monitor API.

Exit codes

MEG_BASE_URL and MEG_API_URL change the two hosts.