GET /v1/batches
List your account’s batches, filter, search and sort them.
Newest first unless sort and order say otherwise. Walk the pages by
sending next_cursor back as cursor, with the same filters and sort,
until it is null. total counts every batch the filters match, on
every page. Each batch carries only the counts stored on it (done,
failed, charged_credits); GET /v1/batches/{id} has the rest.
Parameters
| name |
in |
required |
type or values |
what it is |
cursor |
query |
no |
string |
The next_cursor of the previous page, unchanged. Absent means the first page. |
limit |
query |
no |
integer, 1 to 100; default 50 |
Rows per page, 1 to 100. |
status |
query |
no |
queued, running, done, failed, canceled |
Only batches in this status. Absent means every status. |
engine |
query |
no |
http, browser, auto |
Only batches sent with this engine. auto means the batch left engine to us. |
created_after |
query |
no |
string |
Only batches created at or after this time. RFC 3339. |
created_before |
query |
no |
string |
Only batches created before this time. RFC 3339. |
q |
query |
no |
string |
Only batches whose name contains this text, ignoring case. 1 to 128 characters; % and _ match themselves. |
sort |
query |
no |
created, credits; default created |
created sorts by when the batch was made, credits by the credits it was charged. |
order |
query |
no |
desc, asc; default desc |
|
Answers
| status |
what it means |
200 |
One page of batches. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |
POST /v1/batches
Submit a batch of URLs to scrape in the background.
Asynchronous. The call answers 202 as soon as the batch is stored and
nothing is charged yet: each URL is priced, reserved and settled on its
own when a worker reaches it. Poll GET /v1/batches/{id} for the counts and
GET /v1/batches/{id}/items for the results. The body is the POST /v1
envelope (see v1.yaml) shared by the whole batch, plus urls, and
optionally deadline, webhook_url and crawl. One refused URL is a 400
for the whole batch and nothing is stored. Authorization header only.
Parameters
| name |
in |
required |
type or values |
what it is |
Idempotency-Key |
header |
no |
string |
Your own id for this call, 1 to 128 printable ASCII characters. A second call with the same key inside 24 hours answers 200 with the batch the first call made, and creates and charges nothing. |
Body
| field |
required |
type or values |
what it is |
name |
no |
string |
Your label for the batch. No control characters. |
urls |
yes |
array of string |
|
deadline |
no |
string |
RFC 3339 in UTC, at least 15 minutes and at most 30 days ahead. |
webhook_url |
no |
string |
|
crawl |
no |
object |
Follow links from one page. urls must hold exactly one URL. Each page gives the response_types or schema you sent; links are read from its HTML. |
Answers
| status |
what it means |
200 |
Replay. This Idempotency-Key already created a batch; this is that batch. Nothing new was created and nothing is charged. |
202 |
Batch accepted. Nothing is charged yet. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
402 |
over_plan. Your plan does not include batches (the free plan). Subscribe to a paid plan, or send each page to GET /v1. |
403 |
forbidden_target, one URL is on a site we do not serve; or wrong_scope, an account key (meg_ak_). Batches need a scrape key (meg_sk_). |
413 |
body_too_large. The body was larger than 8 MB. Split the batch. |
415 |
unsupported_media_type. Send Content-Type: application/json, even for an empty {} body. |
422 |
idempotency_mismatch. This Idempotency-Key already created a batch with a different body. Send the same body, or a new key. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |
POST /v1/batches/quote
Price a batch before you send it.
Takes the POST /v1/batches body and answers the most one URL can cost and
the most the whole batch can cost, in credits. It is the same number a
worker reserves for each URL, so a max_credits of each is never too
low at the time of the quote. Nothing is stored or charged.
Body
| field |
required |
type or values |
what it is |
name |
no |
string |
Your label for the batch. No control characters. |
urls |
yes |
array of string |
|
deadline |
no |
string |
RFC 3339 in UTC, at least 15 minutes and at most 30 days ahead. |
webhook_url |
no |
string |
|
crawl |
no |
object |
Follow links from one page. urls must hold exactly one URL. Each page gives the response_types or schema you sent; links are read from its HTML. |
Answers
| status |
what it means |
200 |
The price. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
403 |
forbidden_target. One URL is on a site we do not serve. |
413 |
body_too_large. The body was larger than 8 MB. Split the batch. |
415 |
unsupported_media_type. Send Content-Type: application/json, even for an empty {} body. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy, or unavailable when prices are not loaded yet. Nothing is charged. Send the same request again after the Retry-After seconds. |
GET /v1/batches/{id}
Read a batch’s status and counts.
An id that belongs to another account answers 404, never 403.
Parameters
| name |
in |
required |
type or values |
what it is |
id |
path |
yes |
string |
The batch id POST /v1/batches answered with. |
Answers
| status |
what it means |
200 |
The batch. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
404 |
not_found. No batch with that id in your account. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |
DELETE /v1/batches/{id}
Erase a finished batch’s results and URLs.
Needs an account key (meg_ak_…): erasing cannot be undone, so a
scrape key gets 403 wrong_scope. A scrape key can still cancel.
Deletes every stored result of the batch, every item’s URL and the
request parameters now; engine, proxy_tier and response_types stay.
The batch and its items keep status, counts, credits and erased_at
until we delete them 7 days after the batch ends. Usage and the
credit ledger do not change. Only a done, failed or canceled batch:
a queued or running one answers 409 batch_running, so cancel it
first. Calling it again answers 200 and changes nothing. It cannot be
undone, and download links stop working.
Parameters
| name |
in |
required |
type or values |
what it is |
id |
path |
yes |
string |
The batch id POST /v1/batches answered with. |
Answers
| status |
what it means |
200 |
Erased. The batch as GET /v1/batches/{id} shows it. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
403 |
wrong_scope. A scrape key cannot erase. Use an account key (meg_ak_…) or the dashboard. |
404 |
not_found. No batch with that id in your account. |
409 |
batch_running. The batch is queued or running. Cancel it, then erase it. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |
POST /v1/batches/{id}/cancel
Cancel a batch.
Marks every item that has not started yet as canceled and returns the
credits held for them. Items already running finish and are charged.
Nothing is charged for cancelling. A finished batch answers 200 with
canceled 0. To delete a finished batch’s results and URLs, use
DELETE /v1/batches/{id}.
Parameters
| name |
in |
required |
type or values |
what it is |
id |
path |
yes |
string |
The batch id POST /v1/batches answered with. |
Answers
| status |
what it means |
200 |
Cancelled. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
404 |
not_found. No batch with that id in your account. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |
GET /v1/batches/{id}/items
List a batch’s items, one page at a time.
Walk the pages by sending next_cursor back as cursor until it is null.
A finished item carries result_url, signed for 15 minutes, and
screenshot_url when it has one, signed for 1 hour; ask for the
page again for fresh links.
Parameters
| name |
in |
required |
type or values |
what it is |
id |
path |
yes |
string |
The batch id POST /v1/batches answered with. |
cursor |
query |
no |
string |
The next_cursor of the previous page. Absent means the first page. |
limit |
query |
no |
integer, 1 to 1000; default 100 |
Items per page, 1 to 1000. |
status |
query |
no |
array of pending, published, running, done, failed, canceled, delivered, blocked, target_error, needs_js |
Only items in these states, comma-separated. done is every finished item. delivered is done with a result. blocked (the site refused us), target_error (the site answered 5xx) and needs_js (the page needs the browser) are done and free. Absent means every item. |
Answers
| status |
what it means |
200 |
One page of items. |
400 |
bad_parameter. error.field names the parameter; the message says which values work. An unknown query key is this 400 too. |
401 |
bad_token. The key is missing, not valid, expired, revoked, of an unknown prefix or wrong length, or sent as the api_key query parameter, which no route accepts. |
404 |
not_found. No batch with that id in your account. |
429 |
rate_limited. Too many requests to this endpoint right now. Wait Retry-After seconds, then send the same request again. |
500 |
internal. Our bug. Try again in a minute; if it stays, report the X-Request-Id. |
503 |
busy. A store we need did not answer, or is not configured on this server. Send the same request again after the Retry-After seconds. |