# Send many URLs as one batch > Send many URLs in one call. Poll the batch, page the results, or get a webhook. **TL;DR**: send up to 10,000 URLs in one call, get an id back, poll it, and download each result from a link that is signed for 15 minutes. Nothing is charged at submit. Each URL is priced and charged on its own, when a worker reaches it. Use a batch instead of `POST /v1` when you have many URLs and no one is waiting for the answer. `POST /v1` keeps a connection open per URL; a batch keeps none. The batch API lives at `/v1/batches`, on the API host, `https://api.megalania.io`. Every example below reads only `MEG_API_KEY`. Batches do not live on the scrape host (`https://get.megalania.io`): `POST /v1` is one request in and one answer out, a batch outlives the connection that made it. ## 1. Submit the batch The body is the `POST /v1` envelope, shared by the whole batch, plus `urls`. `url` is not a key here: the URLs are in `urls`. ```sh curl -s -X POST "https://api.megalania.io/v1/batches" \ -H "Authorization: Bearer $MEG_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid)" \ -d '{ "urls": [ "https://lab.megalania.io/paged?page=1", "https://lab.megalania.io/paged?page=2", "https://lab.megalania.io/paged?page=3" ], "engine": "http", "country_code": "de", "webhook_url": "https://hooks.example.com/megalania" }' ``` ```json { "id": "0199a1c2-...-6f31", "item_count": 3, "status": "queued" } ``` | key | values | default | | -- | -- | -- | | `urls` | 1 to 10,000 absolute `http` or `https` URLs | required | | `deadline` | RFC 3339, at least 15 minutes and at most 30 days ahead | none | | `webhook_url` | one absolute URL we `POST` the outcome to at the end | none | | everything else | any `POST /v1` parameter, applied to every URL | — | Things to know: - **Every URL goes through the same check as a sync request.** One bad URL, one unknown parameter, or one address in a private, loopback, link-local, multicast or broadcast range — in `urls` or in `webhook_url` — is a 400 naming the field, and no batch is stored. For a URL the message names the entry number, counting from one, and never the URL itself. Fix the list and send it again. - **`Idempotency-Key` works here too, and a retry is safe.** The first call with a key answers 202 and creates the batch. Any later call with the same key inside 24 hours answers **200** with the same body — the same `id`, `item_count` and `status` — and creates nothing, so a retry that lost its answer still gets the batch back. Send the header: a resubmit without one is a second batch and a second bill. - **A `deadline` only moves work earlier.** Items are scheduled to start five minutes before it. It is not a promise and not a refund. - **A `deadline` does not change the price.** Every page is priced as sync. - **The cap is 10,000.** More is a 400 that says so. Split the list. ### Price the batch first `POST /v1/batches/quote` takes the same body and stores nothing. It answers `{"each": 75, "total": 150}`: the most one URL can cost and the most the whole batch can cost, in credits. It is the number a worker reserves for each URL, so `max_credits` set to `each` is never too low at the time of the quote. ```bash curl -s -X POST "https://api.megalania.io/v1/batches/quote" \ -H "Authorization: Bearer $MEG_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://lab.megalania.io/paged?page=1", "https://lab.megalania.io/paged?page=2"], "proxy_tier": "datacenter"}' ``` ### Crawl a site from one URL Add `crawl` and send exactly one URL. We fetch that page, read its links, and add each new page on the same site as a new item. A crawl is a batch like any other: poll it, page its items and download results the same way. ```sh curl -s -X POST "https://api.megalania.io/v1/batches" \ -H "Authorization: Bearer $MEG_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid)" \ -d '{ "urls": ["https://lab.megalania.io/"], "crawl": { "max_depth": 2, "max_pages": 100, "include_paths": ["/a", "/b", "/b/**", "/private/**"], "exclude_paths": ["/paged"], "respect_robots": true } }' ``` On the Lab this finds `/a`, `/b` and `/b/c`. It skips `/private/x`, because `robots.txt` disallows it. ```json { "id": "0199a1c2-...-6f31", "item_count": 1, "status": "queued" } ``` | key | values | default | | -- | -- | -- | | `crawl.max_depth` | 0 to 10. Links are followed this many clicks away from the first page | 2 | | `crawl.max_pages` | 1 to 10,000. The most items the batch will ever have, the first page included | 100 | | `crawl.include_paths` | Up to 20 path patterns, each starting with `/` and 1 to 256 characters. Only links whose path matches one are followed | none: every path | | `crawl.exclude_paths` | Up to 20 path patterns, same form. A link whose path matches one is never followed, even if it matches `include_paths` | none | | `crawl.include_subdomains` | `true` also follows subdomains of the first page's host (`www.` removed first) | `false` | | `crawl.respect_robots` | `true` skips links the site's `robots.txt` disallows for `User-agent: *` | `false` | A bad value is a 400 `bad_parameter` that names the field, like `crawl.include_paths`. Things to know: - **`item_count` grows while the crawl runs.** It starts at 1. A new item appears when a finished page links to a page the batch does not have yet. - **Only the same site is followed**: the same scheme, host and port as the page the link is on. A link to another host, a subdomain, another port, or a private address is skipped. Only `` links count. With `include_subdomains`, a link to the first page's host or any of its subdomains is followed too, on the same scheme and port. Another site never is. On a shared host such as `github.io`, every site under it is a subdomain, so `include_subdomains` follows all of them. `max_pages` still bounds the crawl and each page is charged. - **Path patterns match the path only**, never the host or the query. `*` matches inside one path segment and `**` matches across segments, so `/blog/*` matches `/blog/a` but not `/blog/a/b`, and `/blog/**` matches both. Every other character matches itself. The first page is always fetched; the patterns decide which links found on pages are followed. - **Path patterns are kept only while the crawl runs.** They are cleared when the batch finishes or is canceled. - **`respect_robots` is free.** We read `robots.txt` once per host per batch and never charge for it. Each read is logged with the batch, at 0 credits. `Allow` and `Disallow` use the longest match, with `*` and `$`. A missing or broken `robots.txt`, or one we cannot reach in a few seconds, allows everything. - **A skipped link costs nothing.** It is never fetched and never becomes an item. - **`robots.txt` and sitemap files are never pages.** A link to `/robots.txt` or to a `sitemap*.xml` (or `.xml.gz`) file never becomes an item, so it is never charged. - **Each page is priced and charged on its own**, like a batch item. Your balance is checked page by page, so a crawl stops adding work you cannot pay for. At most `max_pages` pages are ever charged. - **A page is fetched once per batch.** The same link found on many pages is one item. The part after `#` is ignored. - **Links come from the page's HTML, whatever you ask for.** Send any `response_types`, or a `schema`: each item's result is what you asked for, and the links are still read from the HTML at no extra cost. At most 1,000 links are read from one page. - **Cancel stops the crawl.** No new items are added after a cancel. - Each item in `GET /v1/batches/{id}/items` has a `depth`: 0 for the first page, 1 for the pages it links to, and so on. It is `null` for a batch item. ## 2. Poll the batch ```sh curl -s -H "Authorization: Bearer $MEG_API_KEY" "https://api.megalania.io/v1/batches/$BATCH_ID" ``` ```json { "id": "0199a1c2-...-6f31", "name": null, "status": "running", "params": { "engine": "http", "proxy_tier": null, "response_types": ["html"] }, "webhook_url": null, "item_count": 3, "done_count": 2, "free_count": 0, "failed_count": 0, "deadline": null, "created_at": "2026-09-09T09:12:44Z", "finished_at": null, "erased_at": null, "webhook_status": null, "counts": { "queued": 0, "running": 1, "done": 2, "failed": 0, "blocked": 0, "target_error": 0, "needs_js": 0, "canceled": 0, "charged_credits": 2, "reserved_credits": 1, "returned_credits": 0 } } ``` Each item is in exactly one count, so `counts` adds up to `item_count`. `done` and `done_count` are the items delivered with a result. `blocked` (the site refused us), `target_error` (the site answered 5xx) and `needs_js` (the page needs `engine=browser`) are free and never in `done`; together they are `free_count`. `done_count + free_count + failed_count` plus canceled items is `item_count` once the batch ends. `status` is `queued`, `running`, `done`, `failed` or `canceled`. A batch whose items all ended free is still `done`. Poll every few seconds, or give a `webhook_url` at submit and skip the polling. `params.response_types` lists what each item stores, `["html"]` when you did not set it. `webhook_status` is `pending`, `delivered` or `dead` once we gave up on your receiver, and `null` when the batch has no `webhook_url`. ## 3. Read the items and download the results ```sh curl -s -H "Authorization: Bearer $MEG_API_KEY" \ "https://api.megalania.io/v1/batches/$BATCH_ID/items?limit=100" ``` ```json { "items": [ { "id": "0199a1c3-...-11a0", "position": 1, "depth": null, "url": "https://lab.megalania.io/", "status": "done", "credits_charged": 1, "target_status": 200, "engine": "http", "proxy_tier": "datacenter", "bytes": 18432, "block_kind": null, "document": null, "document_units": 0, "document_truncated": false, "hops": [{"rung": "h:dc", "identity": "firefox", "ms": 412}], "untried": null, "untried_reason": null, "cache": null, "finished_at": "2026-09-09T09:13:05Z", "error": null, "result_url": "https://…/results/…?X-Amz-Expires=900&…", "screenshot_url": null } ], "next_cursor": "1" } ``` Walk the pages by sending `next_cursor` back as `cursor` until it is `null`: ```sh cursor="" while : ; do page=$(curl -s -H "Authorization: Bearer $MEG_API_KEY" \ "https://api.megalania.io/v1/batches/$BATCH_ID/items?limit=100&cursor=$cursor") echo "$page" | jq -r '.items[] | select(.result_url) | .result_url' | while read -r url; do curl -sO "$url"; done cursor=$(echo "$page" | jq -r '.next_cursor // empty') [ -n "$cursor" ] || break done ``` Things to know: - **`result_url` points at our result store's public host**, not at the API, so it is a different address from `https://api.megalania.io`. - **`result_url` is signed for 15 minutes**, and only a finished item has one. Ask for the page again to get a fresh link. It carries its own credentials, so fetch it without your token. It is signed for `GET` only: a `HEAD` on it answers 403. - **`screenshot_url` is signed for 1 hour on every read**, only on a finished item that asked for `screenshot` and got one. The stored result keeps the screenshot's `bytes` and `credits`, never a link, so an old result still gets a working link. It downloads as `screenshot--.jpg`. - **`result_url` downloads a named file**: `batch-result--.`, where the extension follows the result's type (`.md`, `.json`, `.html`, `.txt`). The name never holds the page's URL or host. - **One type is one file of that type.** Several `response_types`, or a `screenshot`, give one JSON object with a key per type (`html`, `markdown`, `network`, ...) and `screenshot`. A `schema` gives the same JSON answer as `GET /v1`. - **`position` is the item's place in the `urls` you sent**, starting at one, and it is what `cursor` holds. The order never changes. - **`limit` is 1 to 1000**, 100 by default. - **`status` picks items**, comma-separated: `pending`, `published`, `running`, `done`, `failed`, `canceled`, and the kinds of `done`: `delivered` (it has a result), and the free ones by their `error`: `blocked` (the site refused us), `target_error` (the site answered 5xx) and `needs_js` (the page needs the browser). For example `?status=blocked,target_error,needs_js,failed` lists what to look at again. A free item has no `result_url`, and its `error` says which. - **`hops` is the routes we tried**, in order, the same JSON array as the `X-Hops` header of `GET /v1`: `rung`, `identity` and `ms` per try. `untried` lists the routes your key may use that we did not try, only on a blocked or `needs_js` answer. `untried_reason` is `budget` when residential routes were unavailable, so the climb skipped residential; a skipped route is never charged. - **`finished_at` is when the item ended**, `null` while it waits or runs. - **`cache` says the item was a copy.** An item served from a recent copy of the page has `"cache": {"hit": true}`. It is `null` otherwise. See [Reuse a recent copy](/guides/cache). - **A `max_credits` below an item's price fails that item for free.** Its `status` is `failed`, and its `error` names `max_credits`. - **A `country_code` no route serves fails the item for free.** Its `error` is `bad_parameter` on `country_code`. [Countries](/guides/countries) lists what each route serves. - **A batch without `country_code` uses your account's default country** from the moment you send it, and pays its premium if it has one. The batch's `params` show that country. See [Countries](/guides/countries#your-accounts-default-country). ## 4. List your batches ```sh curl -s -H "Authorization: Bearer $MEG_API_KEY" \ "https://api.megalania.io/v1/batches?status=done&sort=credits&order=desc&limit=25" ``` ```json { "batches": [ { "id": "0199a1c2-...-6f31", "status": "done", "...": "..." } ], "next_cursor": "118.0199a1c2-...-6f31", "total": 143 } ``` Each batch has the same fields as `GET /v1/batches/{id}`, except `counts`: the list carries only the counts stored on the batch, `done`, `free`, `failed` and `charged_credits`. Read the batch for `queued`, `running`, `blocked`, `target_error`, `needs_js`, `canceled`, `reserved_credits` and `returned_credits`. `total` counts every batch the filters match, not only this page. Send `next_cursor` back as `cursor`, with the same filters and sort, until it is `null`. | key | values | default | | -- | -- | -- | | `status` | `queued`, `running`, `done`, `failed` or `canceled` | every status | | `engine` | `http`, `browser`, or `auto` for a batch that left engine to us | every engine | | `created_after`, `created_before` | RFC 3339 times; after is inclusive, before is not | none | | `q` | 1 to 128 characters the batch `name` contains, ignoring case | none | | `sort` | `created` or `credits` (credits charged) | `created` | | `order` | `desc` or `asc` | `desc` | | `limit` | 1 to 100 | 50 | A cursor belongs to its sort: a cursor from `sort=credits` sent with `sort=created` is a 400 on `cursor`. ## 5. Cancel ```sh curl -s -X POST -H "Authorization: Bearer $MEG_API_KEY" \ "https://api.megalania.io/v1/batches/$BATCH_ID/cancel" ``` ```json { "id": "0199a1c2-...-6f31", "status": "canceled", "canceled": 41, "returned_credits": 3075 } ``` `canceled` is how many items this call stopped, and `returned_credits` the credits held for them that come back to you. Items already running finish and are charged; items that never started are not. While an item still runs, `status` does not change yet; the batch turns `canceled` when that item ends. An item whose credits were held but whose turn had not come gets them back within seconds, when the queue reaches it, so a balance read right after a cancel can still be low for a moment. Cancelling itself costs nothing. ## 6. Erase a finished batch Erase needs an **account key** (`meg_ak_...`), because it cannot be undone. Make one on the Keys page, or erase from the dashboard. ```sh curl -s -X DELETE -H "Authorization: Bearer $MEG_ACCOUNT_KEY" \ "https://api.megalania.io/v1/batches/$BATCH_ID" ``` The answer is the batch, as `GET /v1/batches/{id}` shows it, with `erased_at` set. We delete every stored result of the batch, every item's URL, and the request parameters you sent (headers, cookies, body, steps) now; only `engine`, `proxy_tier` and `response_types` stay. Your balance does not change. Things to know: - **A scrape key cannot erase.** It gets **403** `wrong_scope`. A scrape key can still cancel a batch. - **Only a finished batch.** A `queued` or `running` batch answers **409** `batch_running`. Cancel it first, and erase it once its status is `canceled`. - **It cannot be undone.** Download links stop working, the screenshots are deleted, and each item's `url`, `result_url` and `screenshot_url` are `null` from then on. - **A second call is safe.** It answers 200 and changes nothing. ## 7. The webhook If you sent a `webhook_url`, we `POST` it once when the batch finishes, done or failed. Nothing else is sent: an item finishing on its own is not an event. ```http POST /your/path HTTP/1.1 Content-Type: application/json X-Megalania-Event: batch.completed X-Megalania-Timestamp: 1757410442 X-Megalania-Signature: sha256=6f1c…9b2a { "id": "0199a1c2-...-6f31", "status": "done", "item_count": 3, "done_count": 3, "free_count": 0, "failed_count": 0, "finished_at": "2026-09-09T09:14:02Z" } ``` The body carries no result and no URL. Read the items when it arrives. ### Check the signature An owner gets the webhook secret with `POST /v1/account/webhook-secret/rotate` (account key), or on the dashboard: Connect, Webhooks, Rotate and show. We show a secret once, when it is made; we never show it again. See [Your account, Webhook secret](/guides/account#webhook-secret). The signature is `sha256=` plus the HMAC-SHA256 of `X-Megalania-Timestamp`, a dot, and the exact bytes of the body, keyed with that secret. Sign the raw body, not a re-serialized copy of it, and compare in constant time. For 24 hours after a rotation the old secret signs too, so the header carries two values, newest first: `sha256=,sha256=`. Accept the delivery when any one of them matches. Your receiver keeps working while you swap secrets. Refuse a timestamp more than five minutes away from your own clock. Without that check a POST captured once can be replayed forever: the body of a finished batch never changes. ```python import hashlib, hmac, time def valid(raw: bytes, stamp: str, header: str, secret: str) -> bool: if abs(time.time() - int(stamp)) > 300: return False signed = stamp.encode() + b"." + raw mine = "sha256=" + hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest() return any(hmac.compare_digest(mine, part) for part in header.split(",")) ``` ```ts import { createHmac, timingSafeEqual } from "node:crypto"; export function valid(raw: Buffer, stamp: string, header: string | null, secret: string): boolean { if (Math.abs(Date.now() / 1000 - Number(stamp)) > 300) return false; const mine = Buffer.from("sha256=" + createHmac("sha256", secret).update(`${stamp}.`).update(raw).digest("hex")); return (header ?? "").split(",").some((part) => { const theirs = Buffer.from(part); return theirs.length === mine.length && timingSafeEqual(mine, theirs); }); } ``` From a shell, with the stamp, the raw body and your secret, this prints the value one part of `X-Megalania-Signature` must equal: ```sh STAMP=1757410000 BODY='{"id":"x"}' SECRET=a-secret printf '%s.%s' "$STAMP" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* /sha256=/' ``` Send yourself a signed `test` event to check your receiver end to end: [Your account, Webhooks](/guides/account#webhooks). ### What we retry Answer `2xx` and we stop. Anything else, including a redirect, is a failure: we do not follow redirects, and we give up on a request after 10 seconds. Five deliveries at most, waiting 1 s, 5 s, 25 s and 125 s in between. After the fifth the batch is marked dead and nothing else is sent for it. A delivery can repeat after your `2xx` if our answer was lost, so treat the `id` as the key and make your handler idempotent. The URL must name a public address, at submit and again at delivery time: a webhook to a loopback, private, link-local, multicast or broadcast address is refused. ## What we store, and for how long Two things worth knowing before you send a list: - **A batch and its items are deleted 7 days after the batch ends**: the URLs, the counts and the credits per item, erased or not. Your credit ledger is separate and stays, so your balance still counts what you paid. - **Each result is deleted 7 days after it is saved**, so at the latest 7 days after the batch ends. A batch that runs longer than 7 days loses its oldest results first: fetch them as they finish. - **The URLs of a batch are stored in our database** while the batch runs, until that deletion, or at once when you erase the batch. - **Headers, cookies, a body and steps are kept only while the batch runs.** Once it ends, only `engine`, `proxy_tier` and `response_types` stay, so a retry from the dashboard sends each URL again as a plain GET. - **A `result_url` or `screenshot_url` is a bearer link.** Anyone who holds it can fetch that file until it expires, 15 minutes (1 hour for a screenshot) after we sign it. Treat it like a password: do not paste it into a ticket, a log, or a chat. ## Your batches are yours alone A batch id that belongs to another key answers **404**, never 403, on every route. We do not confirm that somebody else's id exists. ## Errors Same shapes and codes as `/v1`; see [Error codes](/reference/errors). The ones you will meet here: | status | code | what to change | | -- | -- | -- | | 400 | `bad_parameter` | `error.field` names `urls`, `deadline`, `webhook_url`, `cursor`, `limit`, `status`, `engine`, `created_after`, `created_before`, `q`, `sort`, `order`, or the scrape parameter that is wrong | | 401 | `bad_token` | no key, a key in the wrong place, or a wrong, expired or deleted key; the message says which | | 403 | `wrong_scope` | an account key on submit, read or cancel; or a scrape key on erase, which needs an account key | | 404 | `not_found` | no batch with that id under your key | | 409 | `batch_running` | the batch is still queued or running: cancel it, then erase it | | 413 | `body_too_large` | the body was over 8 MB; split the batch | | 503 | `busy` | our store did not answer, or your first submit with this `Idempotency-Key` is still being saved; send the same request with the same key again | One item limit: a page over 10 MiB ends that item with `body_cap`, free of charge. `POST /v1` takes pages up to 50 MB: fetch a bigger page there.