Send many URLs as one batch

Send up to 10,000 URLs in one call, get an id back, poll it, and download each result from a link that is signed for 15 minutes. Nothing is charged at submit. Each URL is priced and charged on its own, when a worker reaches it.

Use a batch instead of POST /v1 when you have many URLs and no one is waiting for the answer. POST /v1 keeps a connection open per URL; a batch keeps none. The batch API lives at /v1/batches, on the API host, https://api.megalania.io. Every example below reads only MEG_API_KEY. Batches do not live on the scrape host (https://get.megalania.io): POST /v1 is one request in and one answer out, a batch outlives the connection that made it.

1. Submit the batch

The body is the POST /v1 envelope, shared by the whole batch, plus urls. url is not a key here: the URLs are in urls.

curl -s -X POST "https://api.megalania.io/v1/batches" \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid)" \
  -d '{
        "urls": [
          "https://lab.megalania.io/paged?page=1",
          "https://lab.megalania.io/paged?page=2",
          "https://lab.megalania.io/paged?page=3"
        ],
        "engine": "http",
        "country_code": "de",
        "webhook_url": "https://hooks.example.com/megalania"
      }'

{ "id": "0199a1c2-...-6f31", "item_count": 3, "status": "queued" }

Things to know:

• Every URL goes through the same check as a sync request. One bad URL, one unknown parameter, or one address in a private, loopback, link-local, multicast or broadcast range — in urls or in webhook_url — is a 400 naming the field, and no batch is stored. For a URL the message names the entry number, counting from one, and never the URL itself. Fix the list and send it again.
• Idempotency-Key works here too, and a retry is safe. The first call with a key answers 202 and creates the batch. Any later call with the same key inside 24 hours answers 200 with the same body — the same id, item_count and status — and creates nothing, so a retry that lost its answer still gets the batch back. Send the header: a resubmit without one is a second batch and a second bill.
• A deadline only moves work earlier. Items are scheduled to start five minutes before it. It is not a promise and not a refund.
• A deadline does not change the price. Every page is priced as sync.
• The cap is 10,000. More is a 400 that says so. Split the list.

Price the batch first

POST /v1/batches/quote takes the same body and stores nothing. It answers {"each": 75, "total": 150}: the most one URL can cost and the most the whole batch can cost, in credits. It is the number a worker reserves for each URL, so max_credits set to each is never too low at the time of the quote.

curl -s -X POST "https://api.megalania.io/v1/batches/quote" \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://lab.megalania.io/paged?page=1", "https://lab.megalania.io/paged?page=2"], "proxy_tier": "datacenter"}'

Crawl a site from one URL

Add crawl and send exactly one URL. We fetch that page, read its links, and add each new page on the same site as a new item. A crawl is a batch like any other: poll it, page its items and download results the same way.

curl -s -X POST "https://api.megalania.io/v1/batches" \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid)" \
  -d '{
        "urls": ["https://lab.megalania.io/"],
        "crawl": {
          "max_depth": 2,
          "max_pages": 100,
          "include_paths": ["/a", "/b", "/b/**", "/private/**"],
          "exclude_paths": ["/paged"],
          "respect_robots": true
        }
      }'

On the Lab this finds /a, /b and /b/c. It skips /private/x, because robots.txt disallows it.

{ "id": "0199a1c2-...-6f31", "item_count": 1, "status": "queued" }

A bad value is a 400 bad_parameter that names the field, like crawl.include_paths.

Things to know:

• item_count grows while the crawl runs. It starts at 1. A new item appears when a finished page links to a page the batch does not have yet.
• Only the same site is followed: the same scheme, host and port as the page the link is on. A link to another host, a subdomain, another port, or a private address is skipped. Only <a href> links count. With include_subdomains, a link to the first page’s host or any of its subdomains is followed too, on the same scheme and port. Another site never is. On a shared host such as github.io, every site under it is a subdomain, so include_subdomains follows all of them. max_pages still bounds the crawl and each page is charged.
• Path patterns match the path only, never the host or the query. * matches inside one path segment and ** matches across segments, so /blog/* matches /blog/a but not /blog/a/b, and /blog/** matches both. Every other character matches itself. The first page is always fetched; the patterns decide which links found on pages are followed.
• Path patterns are kept only while the crawl runs. They are cleared when the batch finishes or is canceled.
• respect_robots is free. We read robots.txt once per host per batch and never charge for it. Each read is logged with the batch, at 0 credits. Allow and Disallow use the longest match, with * and $. A missing or broken robots.txt, or one we cannot reach in a few seconds, allows everything.
• A skipped link costs nothing. It is never fetched and never becomes an item.
• robots.txt and sitemap files are never pages. A link to /robots.txt or to a sitemap*.xml (or .xml.gz) file never becomes an item, so it is never charged.
• Each page is priced and charged on its own, like a batch item. Your balance is checked page by page, so a crawl stops adding work you cannot pay for. At most max_pages pages are ever charged.
• A page is fetched once per batch. The same link found on many pages is one item. The part after # is ignored.
• Links come from the page’s HTML, whatever you ask for. Send any response_types, or a schema: each item’s result is what you asked for, and the links are still read from the HTML at no extra cost. At most 1,000 links are read from one page.
• Cancel stops the crawl. No new items are added after a cancel.
• Each item in GET /v1/batches/{id}/items has a depth: 0 for the first page, 1 for the pages it links to, and so on. It is null for a batch item.

2. Poll the batch

curl -s -H "Authorization: Bearer $MEG_API_KEY" "https://api.megalania.io/v1/batches/$BATCH_ID"

{
  "id": "0199a1c2-...-6f31",
  "name": null,
  "status": "running",
  "params": { "engine": "http", "proxy_tier": null, "response_types": ["html"] },
  "webhook_url": null,
  "item_count": 3,
  "done_count": 2,
  "free_count": 0,
  "failed_count": 0,
  "deadline": null,
  "created_at": "2026-09-09T09:12:44Z",
  "finished_at": null,
  "erased_at": null,
  "webhook_status": null,
  "counts": {
    "queued": 0, "running": 1, "done": 2, "failed": 0, "blocked": 0,
    "target_error": 0, "needs_js": 0, "canceled": 0,
    "charged_credits": 2, "reserved_credits": 1, "returned_credits": 0
  }
}

Each item is in exactly one count, so counts adds up to item_count. done and done_count are the items delivered with a result. blocked (the site refused us), target_error (the site answered 5xx) and needs_js (the page needs engine=browser) are free and never in done; together they are free_count. done_count + free_count + failed_count plus canceled items is item_count once the batch ends.

status is queued, running, done, failed or canceled. A batch whose items all ended free is still done. Poll every few seconds, or give a webhook_url at submit and skip the polling. params.response_types lists what each item stores, ["html"] when you did not set it.

webhook_status is pending, delivered or dead once we gave up on your receiver, and null when the batch has no webhook_url.

3. Read the items and download the results

curl -s -H "Authorization: Bearer $MEG_API_KEY" \
  "https://api.megalania.io/v1/batches/$BATCH_ID/items?limit=100"

{
  "items": [
    {
      "id": "0199a1c3-...-11a0",
      "position": 1,
      "depth": null,
      "url": "https://lab.megalania.io/",
      "status": "done",
      "credits_charged": 1,
      "target_status": 200,
      "engine": "http",
      "proxy_tier": "datacenter",
      "bytes": 18432,
      "block_kind": null,
      "document": null,
      "document_units": 0,
      "document_truncated": false,
      "hops": [{"rung": "h:dc", "identity": "firefox", "ms": 412}],
      "untried": null,
      "untried_reason": null,
      "cache": null,
      "finished_at": "2026-09-09T09:13:05Z",
      "error": null,
      "result_url": "https://…/results/…?X-Amz-Expires=900&…",
      "screenshot_url": null
    }
  ],
  "next_cursor": "1"
}

Walk the pages by sending next_cursor back as cursor until it is null:

cursor=""
while : ; do
  page=$(curl -s -H "Authorization: Bearer $MEG_API_KEY" \
    "https://api.megalania.io/v1/batches/$BATCH_ID/items?limit=100&cursor=$cursor")
  echo "$page" | jq -r '.items[] | select(.result_url) | .result_url' |
    while read -r url; do curl -sO "$url"; done
  cursor=$(echo "$page" | jq -r '.next_cursor // empty')
  [ -n "$cursor" ] || break
done

Things to know:

• result_url points at our result store’s public host, not at the API, so it is a different address from https://api.megalania.io.
• result_url is signed for 15 minutes, and only a finished item has one. Ask for the page again to get a fresh link. It carries its own credentials, so fetch it without your token. It is signed for GET only: a HEAD on it answers 403.
• screenshot_url is signed for 1 hour on every read, only on a finished item that asked for screenshot and got one. The stored result keeps the screenshot’s bytes and credits, never a link, so an old result still gets a working link. It downloads as screenshot-<item id>-<date>.jpg.
• result_url downloads a named file: batch-result-<item id>-<date>.<ext>, where the extension follows the result’s type (.md, .json, .html, .txt). The name never holds the page’s URL or host.
• One type is one file of that type. Several response_types, or a screenshot, give one JSON object with a key per type (html, markdown, network, …) and screenshot. A schema gives the same JSON answer as GET /v1.
• position is the item’s place in the urls you sent, starting at one, and it is what cursor holds. The order never changes.
• limit is 1 to 1000, 100 by default.
• status picks items, comma-separated: pending, published, running, done, failed, canceled, and the kinds of done: delivered (it has a result), and the free ones by their error: blocked (the site refused us), target_error (the site answered 5xx) and needs_js (the page needs the browser). For example ?status=blocked,target_error,needs_js,failed lists what to look at again. A free item has no result_url, and its error says which.
• hops is the routes we tried, in order, the same JSON array as the X-Hops header of GET /v1: rung, identity and ms per try. untried lists the routes your key may use that we did not try, only on a blocked or needs_js answer. untried_reason is budget when residential routes were unavailable, so the climb skipped residential; a skipped route is never charged.
• finished_at is when the item ended, null while it waits or runs.
• cache says the item was a copy. An item served from a recent copy of the page has "cache": {"hit": true}. It is null otherwise. See Reuse a recent copy.
• A max_credits below an item’s price fails that item for free. Its status is failed, and its error names max_credits.
• A country_code no route serves fails the item for free. Its error is bad_parameter on country_code. Countries lists what each route serves.
• A batch without country_code uses your account’s default country from the moment you send it, and pays its premium if it has one. The batch’s params show that country. See Countries.

4. List your batches

curl -s -H "Authorization: Bearer $MEG_API_KEY" \
  "https://api.megalania.io/v1/batches?status=done&sort=credits&order=desc&limit=25"

{ "batches": [ { "id": "0199a1c2-...-6f31", "status": "done", "...": "..." } ], "next_cursor": "118.0199a1c2-...-6f31", "total": 143 }

Each batch has the same fields as GET /v1/batches/{id}, except counts: the list carries only the counts stored on the batch, done, free, failed and charged_credits. Read the batch for queued, running, blocked, target_error, needs_js, canceled, reserved_credits and returned_credits. total counts every batch the filters match, not only this page. Send next_cursor back as cursor, with the same filters and sort, until it is null.

A cursor belongs to its sort: a cursor from sort=credits sent with sort=created is a 400 on cursor.

5. Cancel

curl -s -X POST -H "Authorization: Bearer $MEG_API_KEY" \
  "https://api.megalania.io/v1/batches/$BATCH_ID/cancel"

{ "id": "0199a1c2-...-6f31", "status": "canceled", "canceled": 41, "returned_credits": 3075 }

canceled is how many items this call stopped, and returned_credits the credits held for them that come back to you. Items already running finish and are charged; items that never started are not. While an item still runs, status does not change yet; the batch turns canceled when that item ends. An item whose credits were held but whose turn had not come gets them back within seconds, when the queue reaches it, so a balance read right after a cancel can still be low for a moment. Cancelling itself costs nothing.

6. Erase a finished batch

Erase needs an account key (meg_ak_...), because it cannot be undone. Make one on the Keys page, or erase from the dashboard.

curl -s -X DELETE -H "Authorization: Bearer $MEG_ACCOUNT_KEY" \
  "https://api.megalania.io/v1/batches/$BATCH_ID"

The answer is the batch, as GET /v1/batches/{id} shows it, with erased_at set. We delete every stored result of the batch, every item’s URL, and the request parameters you sent (headers, cookies, body, steps) now; only engine, proxy_tier and response_types stay. Your balance does not change.

Things to know:

• A scrape key cannot erase. It gets 403 wrong_scope. A scrape key can still cancel a batch.
• Only a finished batch. A queued or running batch answers 409 batch_running. Cancel it first, and erase it once its status is canceled.
• It cannot be undone. Download links stop working, the screenshots are deleted, and each item’s url, result_url and screenshot_url are null from then on.
• A second call is safe. It answers 200 and changes nothing.

7. The webhook

If you sent a webhook_url, we POST it once when the batch finishes, done or failed. Nothing else is sent: an item finishing on its own is not an event.

POST /your/path HTTP/1.1
Content-Type: application/json
X-Megalania-Event: batch.completed
X-Megalania-Timestamp: 1757410442
X-Megalania-Signature: sha256=6f1c…9b2a

{
  "id": "0199a1c2-...-6f31",
  "status": "done",
  "item_count": 3,
  "done_count": 3,
  "free_count": 0,
  "failed_count": 0,
  "finished_at": "2026-09-09T09:14:02Z"
}

The body carries no result and no URL. Read the items when it arrives.

Check the signature

An owner gets the webhook secret with POST /v1/account/webhook-secret/rotate (account key), or on the dashboard: Connect, Webhooks, Rotate and show. We show a secret once, when it is made; we never show it again. See Your account, Webhook secret.

The signature is sha256= plus the HMAC-SHA256 of X-Megalania-Timestamp, a dot, and the exact bytes of the body, keyed with that secret. Sign the raw body, not a re-serialized copy of it, and compare in constant time.

For 24 hours after a rotation the old secret signs too, so the header carries two values, newest first: sha256=<new>,sha256=<old>. Accept the delivery when any one of them matches. Your receiver keeps working while you swap secrets.

Refuse a timestamp more than five minutes away from your own clock. Without that check a POST captured once can be replayed forever: the body of a finished batch never changes.

import hashlib, hmac, time

def valid(raw: bytes, stamp: str, header: str, secret: str) -> bool:
    if abs(time.time() - int(stamp)) > 300:
        return False
    signed = stamp.encode() + b"." + raw
    mine = "sha256=" + hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(mine, part) for part in header.split(","))

import { createHmac, timingSafeEqual } from "node:crypto";

export function valid(raw: Buffer, stamp: string, header: string | null, secret: string): boolean {
  if (Math.abs(Date.now() / 1000 - Number(stamp)) > 300) return false;
  const mine = Buffer.from("sha256=" + createHmac("sha256", secret).update(`${stamp}.`).update(raw).digest("hex"));
  return (header ?? "").split(",").some((part) => {
    const theirs = Buffer.from(part);
    return theirs.length === mine.length && timingSafeEqual(mine, theirs);
  });
}

From a shell, with the stamp, the raw body and your secret, this prints the value one part of X-Megalania-Signature must equal:

STAMP=1757410000 BODY='{"id":"x"}' SECRET=a-secret
printf '%s.%s' "$STAMP" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* /sha256=/'

Send yourself a signed test event to check your receiver end to end: Your account, Webhooks.

What we retry

Answer 2xx and we stop. Anything else, including a redirect, is a failure: we do not follow redirects, and we give up on a request after 10 seconds. Five deliveries at most, waiting 1 s, 5 s, 25 s and 125 s in between. After the fifth the batch is marked dead and nothing else is sent for it.

A delivery can repeat after your 2xx if our answer was lost, so treat the id as the key and make your handler idempotent.

The URL must name a public address, at submit and again at delivery time: a webhook to a loopback, private, link-local, multicast or broadcast address is refused.

What we store, and for how long

Two things worth knowing before you send a list:

• A batch and its items are deleted 7 days after the batch ends: the URLs, the counts and the credits per item, erased or not. Your credit ledger is separate and stays, so your balance still counts what you paid.
• Each result is deleted 7 days after it is saved, so at the latest 7 days after the batch ends. A batch that runs longer than 7 days loses its oldest results first: fetch them as they finish.
• The URLs of a batch are stored in our database while the batch runs, until that deletion, or at once when you erase the batch.
• Headers, cookies, a body and steps are kept only while the batch runs. Once it ends, only engine, proxy_tier and response_types stay, so a retry from the dashboard sends each URL again as a plain GET.
• A result_url or screenshot_url is a bearer link. Anyone who holds it can fetch that file until it expires, 15 minutes (1 hour for a screenshot) after we sign it. Treat it like a password: do not paste it into a ticket, a log, or a chat.

Your batches are yours alone

A batch id that belongs to another key answers 404, never 403, on every route. We do not confirm that somebody else’s id exists.

Errors

Same shapes and codes as /v1; see Error codes. The ones you will meet here:

One item limit: a page over 10 MiB ends that item with body_cap, free of charge. POST /v1 takes pages up to 50 MB: fetch a bigger page there.