# Fetch, extract and lists

> Every parameter of POST /v1, what you get back, sessions, identities and the free lookups.

**TL;DR**: one endpoint, two ways in. `POST /v1` with a JSON object is the
main one. `GET /v1?url=...` is the paste-a-URL one. New here? Start with
[the quickstart](/start/quickstart). [Error codes](/reference/errors) has every
`error.code`, [Batches](/guides/batches) sends many URLs in one call,
[Monitors](/guides/monitors) watch pages on a schedule,
[Map a site](/guides/map) lists a site's URLs in one call, and
[MCP for agents](/guides/mcp) connects Claude, ChatGPT, Cursor or VS Code.
Every parameter, its values and its default are in
[the API reference](/reference/api), generated from the API contract.

The scraping host is `https://get.megalania.io`.

Send your key (`meg_sk_...`) as `Authorization: Bearer <your key>`, on every route. That
is the only way in. A request with `api_key` in the query is a 401
`bad_token`, even when it also carries the header, and it is never charged.

A key in a URL is written down by everything the URL passes: your shell
history, your browser history, every proxy and CDN access log on the way, and
any `Referer` your page sends. We cannot clean those for you, so we do not
accept one. If a URL carrying your key was shared, revoke the key and make a new one.

Send `Idempotency-Key: <your own id>` on any call you might retry: `/v1`,
`POST /v1/batches` and checkout. It is 1 to 128 printable ASCII characters, and
we keep it for 24 hours. On `/v1` a repeat is a 409 `replayed` that names the
first call's `X-Request-Id`, never a second charge. On `POST /v1/batches` and
checkout a repeat returns the first answer. [Error codes](/reference/errors) has the
rules.

## `QUERY /v1`, the standards path

`QUERY /v1` takes the same JSON envelope as `POST /v1`, on the same code path.
Everything is the same: the parameters, the errors, `Idempotency-Key`, the
price. Only the method word changes.

It exists because `QUERY` is the right method for this call. RFC 10008
(Proposed Standard, June 2026) added it for a request that carries a body and
is safe and idempotent — which a render is: it reads a page and changes nothing
at the target.

**`POST` stays the example here.** Browsers are still evaluating `QUERY`, and
an older WAF, corporate proxy or load balancer in front of you may refuse a
method it does not know, usually as a 405 or a 501 that never reaches us. Try
`QUERY` from a server-side client, keep `POST` as the fallback, and use `POST`
wherever a browser or someone else's proxy is in the path.

```sh
curl --request QUERY \
     --header "Authorization: Bearer $MEG_API_KEY" \
     --header "Content-Type: application/json" \
     --data '{"url": "https://docs.megalania.io/start/quickstart"}' \
     "https://get.megalania.io/v1"
```

Every answer from `/v1` carries `Cache-Control: no-store`, on all three
methods. A render is your live data and must never be handed to you from a
cache your side keeps, such as a proxy of yours. This matters most on `QUERY`,
whose answers a cache may keep by default, but it is on every answer so no
method is the exception. Our own short page cache is described in
[Reuse a recent copy](/guides/cache).

## The JSON envelope

```sh
curl --header "Authorization: Bearer $MEG_API_KEY" \
     --header "Content-Type: application/json" \
     --data '{
       "url": "https://example.com/search?q=shoes&sort=price",
       "country_code": "US",
       "method": "POST",
       "content_type": "application/x-www-form-urlencoded",
       "body": "cT1zaG9lcw=="
     }' \
     "https://get.megalania.io/v1"
```

One JSON object. Its keys are the parameter names of the `GET`, plus three
that exist only here:

| key | values | default |
| -- | -- | -- |
| `method` | `GET` or `POST` | `GET` |
| `body` | the target's own request body, base64 | none |
| `content_type` | the media type of `body`, for example `application/json` | none |

`body` and `content_type` need `method: POST`; on a `GET` they are a 400, so a
body is never dropped silently after you were charged.

Every value is a string, except `only_main_content`, which also takes JSON
`true` or `false`; `timeout`, `max_credits` and `max_age`, which also take a
whole JSON number; and `headers`, a list of strings, each `name:value`. Any other type is a 400 naming the key.
Nothing of ours is in the query string on this route, so a target URL keeps
its own query string with no encoding at all, and an unknown key is the same
400 it is on the `GET`. The envelope is at most 1 MB; over that is a 413.

There is no query string on this route. `POST /v1?url=...` is a 400, so a
parameter you left in the URL is never quietly dropped after you were charged.

The token is the `Authorization` header here too: we verify it before we read
a single byte of the body.

Leave `engine` and `proxy_tier` out and we pick the cheapest way to fetch the
page for you: plain HTTP on a datacenter exit first, a residential exit when
the site blocks us. You pay the price of the route that delivers the page, and
only when the page comes back. A page that builds itself in JavaScript needs
`engine=browser`, which sends it to a real browser (Chrome, datacenter by
default): 15 credits and up. `wait_until` other than `load` does the
same. Add `proxy_tier=residential` to run the browser on residential IPs (32
credits and up). [Engines and proxy tiers](/guides/engines) explains the climb.
The browser takes GET only, without `session_id` or `identity`. Try it on the
Lab: `https://lab.megalania.io/js?seed=7` is empty over HTTP and shows
"seed 7" with `engine=browser`.

`max_credits` (1 to 32767) is the most one page may cost. We skip any route
above it. If no route is left, the answer is a free 400
`max_credits_too_low`, before any attempt. A page whose bytes cost more is
cut at the bytes `max_credits` pays for: a 200 with `X-Truncated: max_credits`.

`screenshot=viewport` or `screenshot=full` adds a JPEG of the page as the
browser rendered it: the visible window, or the whole page up to 16,384
pixels tall. It needs the browser, so it sends the request there;
`engine=http` with `screenshot` is a free 400. The answer becomes
`multipart/mixed` with one more part, `screenshot`:
`{"url": "...", "expires_at": "...", "bytes": n, "credits": 1}`. The link
works for 1 hour, for anyone who holds it, and the image is deleted after 7
days. `X-Credits-Charged` is the page alone; the part's `credits` is what the
screenshot adds: 1 for `viewport`, 2 for `full`, charged only once the image
is stored and its link is in the part. If the screenshot could not be made or
stored, the part says `{"error": "screenshot_failed", "credits": 0}` and
nothing is charged for it. A blocked page is free, screenshot included.
Batches take `screenshot` too: the item's result holds the screenshot part
with `bytes` and `credits`, and each read of the batch's items signs a fresh
`screenshot_url` that works for 1 hour.

### Page actions

`actions` runs steps in the browser after the page loads and `wait_until` is
met, in order. Then we read the page and take any screenshot. Use it to click
"load more", fill a search box, or scroll a list.

```
actions=click:.more|wait_for:.item-2|type:#q=shoes|press:Enter|wait_for:.results
```

Entries are `kind:argument`, joined by `|`. At most 20:

| entry | what it does |
| -- | -- |
| `wait:<ms>` | waits 1 to 10,000 ms |
| `wait_for:<css>` | waits up to 10 s for a visible element that matches |
| `click:<css>` | clicks the first visible match, with a real mouse event; waits up to 5 s for it |
| `type:<css>=<text>` | focuses the first visible match and types the text, 1 to 1,024 bytes |
| `press:<key>` | one key: `Enter`, `Tab`, `Escape`, `Backspace`, `Space`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `PageUp`, `PageDown`, `Home` or `End` |
| `scroll:<where>` | `bottom`, `top`, or a number of pixels down |

Try it on the Lab: `https://lab.megalania.io/scroll` sends ten items, and each
scroll to the bottom adds ten more.

```
actions=scroll:bottom|wait_for:#item-20|scroll:bottom|wait_for:#item-30
```

- All actions together get at most 30 s, inside `timeout`. A click that opens
  a new page is followed: the status and the HTML are the new page's.
- A selector is at most 1,024 bytes. `|` ends an entry, so a selector or a
  text cannot hold it (`[lang|=en]` cannot be sent). Selectors do not reach
  into iframes or shadow roots.
- Send `type` by `POST /v1`, so the text is not in a URL.
- `actions` needs the browser, so it sends the request there; `engine=http`
  with `actions` is a free 400 naming `actions`.
- A page with actions costs 3 credits more, charged once with the page. See
  [Credits and pricing](/guides/credits).
- If an action cannot run (no visible match in time, or the 30 s ran out),
  the answer is a free 422 `action_failed`. `error.field` is `actions` and the
  message names the entry's index (0 is the first). It never repeats your
  selector or text, and it carries no page, cookies or target status.
- If every action ran and the page then blocks us, the answer is a free 422
  `blocked`, as always. If an action failed, the answer is `action_failed`,
  even on a blocked page. The [failure cap](/reference/errors#actions-and-the-failure-cap) charges
  failures above 30% an hour.
- Batches take `actions` too, with the same prices and the same failure cap.

### Network capture

`network=true` also returns the JSON and text the page itself loaded with
XHR or `fetch`: an app's own API answers, often cleaner than the HTML. It
runs from the page load through any actions, until we read the page.

```
GET /v1?url=https://shop.example.com/list&network=true&network_url_globs=*/api/*
```

The answer is `multipart/mixed` with one more part, `network`, in
`application/x-ndjson`: one JSON object per response, one per line.

```
{"url":"https://shop.example.com/api/items?page=1","method":"GET","status":200,"content_type":"application/json","bytes":5120,"truncated":false,"body":"{\"items\":[...]}"}
```

- `body` is UTF-8 text. A body that is not text is base64, with
  `"encoding": "base64"`.
- `bytes` is the body's size. A body over 2 MiB is not kept: `body` is empty,
  `truncated` is `true`, and `bytes` is its encoded size on the wire
  (compressed, headers included).
- Without `network_url_globs` we capture JSON, NDJSON and text responses.
  `network_url_globs` keeps only the responses whose full URL matches one of
  its patterns, of any content type: a comma list of at most 10, 1,024 bytes
  in all, where `*` matches anything and `?` one character. It needs
  `network=true`.
- Caps: at most 200 responses, 1 MiB per body (then `truncated` is `true`),
  8 MiB in all, and URLs up to 8 KiB. When a cap cut the list, the answer has
  `X-Network-Truncated: true`. It is also set when a response we would keep
  was still loading when we read the page. A request that never got an
  answer, or a body type we would not keep, does not set it.
- Control characters in captured bodies are JSON-escaped and count toward
  the response size.
- Only the page's own XHR and `fetch` are captured. Requests from web
  workers and service workers are not.
- We return no request or response headers, no cookies and no request
  bodies. Captured URLs can carry the page's own tokens in their query
  strings: they are yours, and we never log or store them.
- It adds no credits: the page's bytes are already metered. See
  [Credits and pricing](/guides/credits).
- `network` needs the browser, so it sends the request there; `engine=http`
  with `network=true` is a free 400 naming `network`. `network` with `schema`
  is a free 400 naming `network`.
- A blocked page is free, and carries no network part.
- Batches take `network` too: the item's result holds the network part.

Some fields are not supported yet, and each answer is a free 400
`bad_parameter`:

- `network` in `response_types`: the error names `response_types`. Send
  `network=true` instead.
- `steps=` as a key of its own: the error says it is not a parameter.

`country_code` defaults to
whatever the deployment is configured with, `US` on the dev cluster.
[Countries](/guides/countries) lists what each route serves, and `GET /v1/geos`
gives it live. A forced route that does not serve the country answers a free
400 on `country_code`. `X-Geo` says which country the request exited from.

## What you get back

`response_types` decides the shape. One type is the body itself, with its own
content type. Several is `multipart/mixed`, one part per type, in the order you
asked for them.

| type | content type | what it is | credits |
| -- | -- | -- | -- |
| `html` | `text/html; charset=utf-8` | the page, the default | in `base` |
| `markdown` | `text/markdown; charset=utf-8` | the same page as Markdown | 0 |
| `text` | `text/plain; charset=utf-8` | the same content as plain text | 0 |
| `links` | `application/json` | every `http(s)` link on the page, with its anchor text | 0 |
| `metadata` | `application/json` | title, description, language, canonical, OpenGraph and JSON-LD | 0 |
| `page_data` | `application/json` | the data the page embeds: JSON-LD and named JSON state such as `__NEXT_DATA__` | 0 |
| `compact_html` | `text/html; charset=utf-8` | the page's HTML without scripts, styles and noise; CSS selectors still match | 0 |
| `inspect` | `application/json` | the page's structure: repeated blocks with ready `extract` rules, headings, links, forms and embedded data | 0 |
| `json` | `application/json` | the fields your `extract` rules read | 1 |

The credits column is the dev prices, and every number in it is a row of the
`billing_price` table rather than anything built into the service. Ask for
several types and you pay for each of them on top of the base request.

### Documents: PDF, Word, PowerPoint and Excel

When the `url` answers with a PDF, DOCX, PPTX or XLSX file, you get the same
`response_types` as for a page. We read the file's own bytes to know what it
is, not its name or its `Content-Type`. `markdown` keeps headings, lists,
tables and links. `text`, `links`, `metadata`, `json` with `extract` rules, and
`schema` all work on it. `html` is the file turned into HTML, not the raw
bytes. Batches and the MCP `scrape` tool take documents the same way.

| file | priced per | dev credits |
| -- | -- | -- |
| PDF | page | 1 |
| PPTX | slide | 1 |
| DOCX | 3,000 characters of text, rounded up | 1 |
| XLSX | 1,000 non-empty cells, rounded up | 1 |

This is on top of the base request. We count the pages after the file
arrives. If your balance cannot cover them, you get a 402
`insufficient_credits`; if your `max_credits` cannot, a 400
`max_credits_too_low`. Either way nothing is sent and nothing
is charged, and the request counts as a free failure.

Limits: 500 pages (or 500 slides, 1,500,000 characters, 500,000 cells), 20,000
characters of text per page or slide, and 64 MiB once the file is unpacked.
Past the page or text limit we convert the first part and say so in
`X-Document-Truncated: true`. Past the unpacked size it is a free 422
`convert_cap`. We read each file in a separate process with its own memory and
time limit. A scanned PDF with no text layer, a file that needs a password, a
damaged file, or one that runs past those limits is a free 422
`document_unreadable`. We do not run OCR yet. A screenshot of a document is not
charged.

The answer carries `X-Document: pdf` (or `docx`, `pptx`, `xlsx`) and
`X-Document-Units`: the pages, slides, characters or cells we counted. Document
content is never logged.

### Structured extraction by schema

Send `response_types=json` with a `schema` instead of `extract`, and a
language model fills your JSON Schema from the page's main content:

```sh
curl -s -H "Authorization: Bearer $MEG_API_KEY" -G "https://get.megalania.io/v1" \
  --data-urlencode "url=https://lab.megalania.io/p/1" \
  --data-urlencode "response_types=json" \
  --data-urlencode 'schema={"type":"object","additionalProperties":false,"required":["title"],"properties":{"title":{"type":"string"},"price":{"type":"number"}}}'
```

```json
{"data": {"title": "Blue mug", "price": 9.0}, "credits": 26, "tokens": 7200}
```

- `schema` is one JSON Schema object of at most 8 KB, with `type: object`
  at the top and `additionalProperties: false` on every object. It may use
  `type`, `properties`, `required`, `items`, `enum`, `const`, `anyOf`,
  `format`, `description` and `title`.
- `prompt` (at most 2 KB) adds your instructions, for example "the main
  product, not the recommendations".
- `extract_model` is `fast`, `default` or `non_china`, never a vendor name.
  We never use a model of China origin, whatever its host. Today all three
  aliases run claude-haiku-4-5.
- `X-Credits-Charged` and `credits` in the answer are the same number: the
  whole request, the page plus the extraction. `llm_credits` is the
  extraction's part: 2, plus a rate per started 1,000 weighted tokens, where
  weighted tokens are input + 5 × output (`tokens`). [Credits and
  pricing](/guides/credits) says where the rates live.
- An answer that does not match the schema is a `422 bad_extract`. The
  extraction is free, the page is still charged, and the body carries the
  page as markdown so you can see what the model saw. After 20 of them on
  your account in a UTC day, the next extraction is a free `429
  llm_token_cap` until 00:00 UTC.
- If your balance covers the page but not the extraction, the answer is a
  `402 insufficient_credits`: the page is charged, the extraction is free,
  and the body carries the page as markdown.
- `screenshot` does not go with `schema`: that is a free 400 naming
  `screenshot`.
- Limits: 96 KB of markdown per request (a longer page is cut, with a
  marker), 8,192 output tokens, and a daily allowance of weighted tokens per
  key and per account (1M in the beta). Over it, the answer is a free `429
  llm_token_cap`.
- Before the model runs, we reserve the most it can cost: one input token
  per byte we send, plus 8,192 output tokens. You are charged what the model
  really used, never more than that reserve. With `max_credits`, the reserve
  must fit under it, or the extraction is skipped, free.
- **Your page content leaves us.** The page's main content, your schema and
  your prompt go to the model provider: Anthropic in the US, never with your account,
  key or who you are. Anthropic may keep it for a time under its own terms. This
  happens only when you send `schema`. See [Privacy](/guides/privacy).
- Batches and crawls take `schema` too: each item's result is the JSON answer,
  charged the same way as one page.

### Reading several types at once

Parts come in the order you listed them in `response_types`. The boundary is
in the `Content-Type` header of the response. Each part has its own
`Content-Type` and a `Content-Disposition: form-data; name="<type>"`, where
`<type>` is the type's lowercase name. A type with nothing in it,
such as `links` on a page with no links, still gets its part, with an empty
body. Save the headers and the body:

```sh
curl -sS -D head.txt -o body.bin -H "Authorization: Bearer $MEG_API_KEY" \
     "https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown,links,metadata"
```

Then split the body with the Python standard library:

```python
import json
import re

head = open("head.txt", encoding="latin-1").read()
ctype = [l.split(":", 1)[1].strip() for l in head.splitlines()
         if l.lower().startswith("content-type:")][-1]
boundary = ("--" + ctype.split("boundary=", 1)[1]).encode()
body = open("body.bin", "rb").read()
assert body.endswith(boundary + b"--\r\n"), "cut short: do not trust the last part"

parts = {}
for chunk in body.split(boundary)[1:-1]:
    part_head, _, data = chunk.partition(b"\r\n\r\n")
    name = re.search(rb'name="([a-z_]+)"', part_head).group(1).decode()
    parts[name] = data[:-2]

markdown = parts["markdown"].decode()
links = json.loads(parts["links"] or b"[]")
metadata = json.loads(parts["metadata"] or b"{}")
print(sorted(parts), len(links), metadata.get("title"))
```

One type is never multipart: the body is the part itself.

`markdown` is the page converted for you: headings, links, images, lists,
tables, code. `script`, `style` and hidden elements are always dropped. With
`only_main_content` (the default), `nav`, `aside`, form controls and a page
`header` or `footer` are dropped too; a `header` or `footer` inside the article
is kept. With `only_main_content=false` they all stay.

On a product page, `markdown` keeps what a shopper needs. A price shown only
to sighted users (the visible text is `aria-hidden`) comes from the label of
its group, once. Option buttons become one line, for example
`Size: XS, S, M (sold out)`. A block the page repeats for mobile and desktop
shows once. Collapsed panels (`<details>`, an accordion a button opens) are
kept. When the page has exactly one JSON-LD `Product`, the markdown ends with
`Product data from the page (JSON-LD):` and its name, brand, price, availability,
rating, SKU and variant count. `compact_html` keeps the same price, options
and collapsed panels; it has no JSON-LD block.

`markdown`, `text` and `compact_html` are the target page's own content. If
you render them as HTML, sanitize them first: they can contain links and markup
the page author chose, and `compact_html` keeps forms, `id`, `name` and remote
images. For an LLM, treat page text as untrusted input: a page can hide
instructions in text that users do not see. For price monitoring, use
markdown or metadata: text output has no strike marker, so an old price and a
new price look alike.

A page that shows its content only after JavaScript runs is rendered in a real
browser. Leave `engine` out and the ladder moves to the browser on its own when
your plan and `max_credits` allow it. When it cannot, the empty shell comes back
free, marked `X-Hint: needs_js`; send `engine=browser` to render it.

Ask for several types in one request, and they come from one read of the
page. You pay for one fetch:

```sh
curl -H "Authorization: Bearer $MEG_API_KEY" \
     "https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown,text,links,metadata"
```

- `text` is the `markdown` content with no markup. Paragraphs stay apart.
- `links` is every `http(s)` link, absolute, once each, with its anchor text.
- `metadata` is the requested URL, title, description, language, canonical, the
  OpenGraph fields and every JSON-LD object on the page, as it was. When the
  page names its next page (`rel="next"`, or one "next" link in a pagination
  block), `next_page` holds that link, absolute, when it stays on the page's
  own site. We never build it from a pattern. On a
  product page, the JSON-LD often holds the price and the specs more reliably
  than the page text.
- `page_data` is the data the page carries for its own scripts:
  `{"json_ld": [...], "state": {"<script id>": ...}}`. `json_ld` is every
  JSON-LD object; `state` is every `<script type="application/json">` with an
  `id`, such as `__NEXT_DATA__` or `__NUXT_DATA__`, parsed. A listing page
  often holds every item there, and only some of them in the visible HTML.
  Ask for `markdown,page_data` to get both from one fetch. It is untrusted
  page content, like the rest.
- `compact_html` is the page's HTML made small for an LLM or a selector
  test. We remove scripts, styles, comments, `svg`, `canvas`, `iframe`,
  `link` and `meta`, hidden elements (`hidden`, `aria-hidden="true"`, inline
  `display:none`), empty elements and every attribute except `id`, `class`,
  `href`, `src`, `alt`, `title`, `name`, `type`, `value`, `datetime`, `lang`,
  `role`, `aria-label`, `itemprop`, `itemtype`, `data-testid` and
  `data-test` (and `content` next to `itemprop`). `href` and `src` are made
  absolute; a `javascript:`, `data:` or other non-http link is removed.
  Whitespace is collapsed, except inside `pre` and `textarea`. The `title`
  stays first. `id` and `class` stay, so a selector you test on the raw page
  still matches. It is cut at about 512 KiB, at an element boundary with every tag
  closed, and then the answer has `X-Compact-Truncated: true`. Like
  `markdown`, it is the main content unless you send
  `only_main_content=false`.
- `inspect` is the page's structure, to help you write `extract` rules
  without reading the page: `{"title", "containers", "headings", "links",
  "forms", "data"}`. Each container is a block that repeats (a product card,
  a search result) with its `selector`, its `count`, a short `sample`, a
  ready `extract` value to send as the `extract` parameter, and the `extract`
  of each field inside it. `data` lists the JSON scripts the page embeds, their
  key paths and an `extract` that reads them with `@json`. It never
  holds the page's text beyond short samples. It reads the whole page, so
  `only_main_content` does not change it. The MCP `inspect` tool returns the
  same object. When `inspect` is the only output and the page shows no
  structure, the answer is a free `422 convert_cap` on field `engine`: the page
  may build its content with JavaScript, so send `engine=browser`. When the
  site answers 4xx, `inspect` answers `200` with `X-Target-Status`, the
  structure as found and a `note` such as "The site answered 404, so there is
  little page structure to inspect.", charged like any answer under 500.

`markdown` and `text` hold the page's main content by default. Some pages keep
a price or a spec table outside it. Send `only_main_content=false` to get the
whole page instead:

```sh
curl -H "Authorization: Bearer $MEG_API_KEY" \
     "https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown&only_main_content=false"
```

`only_main_content` is `true` or `false` in any case, and `true` is the
default. Anything else is a 400 naming `only_main_content`. `false` changes only `markdown`,
`text` and `compact_html`, so it needs one of them in `response_types`; without them it is also a
400.

`markdown`, `text`, `links`, `metadata`, `page_data`, `compact_html`, `inspect` and `json` have size limits:

- The page we convert can be at most 8 MiB of HTML.
- The converted output has a budget. The budget grows with the size of the
  page, so a normal page never comes near it.
- A page over either limit, or one our HTML parser cannot read (for example,
  nested too deep), is a 422 `convert_cap` with `error.field`
  `response_types`. You are not charged.
- The converted types of one request share one budget. Asking for
  `compact_html` next to `markdown` can turn a very large page into a free
  `convert_cap` where `markdown` alone would have fit.
- To fix it, ask for fewer `response_types`, or ask for `html` and convert
  the page yourself. `html` has no such limit.

`json` reads fields out of the page with your own selectors, in `extract`:

```
extract=title:h1|price:.price|links:a@href*
```

One rule is `<name>:<css>`. At most 32 rules and 4096 bytes, joined by `|`. Add
`@attr` to read an attribute instead of the text, and a trailing `*` to get
every match as a list instead of the first one. That example answers:

```json
{"title": "Widgets", "price": "$9.99", "links": ["/a", "/b"]}
```

To read a script's JSON (for example `__NEXT_DATA__`) without the raw page, end the
rule with `@json(path)`: the path is dot-separated keys and array indexes, and
the value comes back as JSON, not text. With `*`, an array gives its elements.
Same price as any `json` rule.

```
extract=products:script#__NEXT_DATA__@json(props.pageProps.products)*
```

A script that is not JSON, or a path that is not there, is a miss.

When each element is a large object, name only the parts you need in `{}`
after the path. Each field is a dot path inside the element; a key that is
not there is `null`, so every row has the same keys. `!key` drops the
elements whose key is true (also a non-empty string, a non-zero number, or a
non-empty list or object); it is the only filter. Fields and `!key` count
toward the 32 rules.

```
extract=items:script#__NEXT_DATA__@json(props.pageProps.items)*{!isSponsored|name:name|price:price.current|link:url}
```

```json
{"items": [{"name": "Oak lamp", "price": "$29.00", "link": "/p/1"}]}
```

For a list of cards, wrap the fields in `{}` after the container selector.
Each match of the container gives one object, and the fields look only inside
it. A field that is missing in one card is `null` in that card, so rows never
shift. On the Lab's cards page, `https://lab.megalania.io/cards`, the second
card has no price:

```html
<div class="card"><h2>Blue mug</h2>
<span class="price">$9</span>
<a href="/p/1">View mug</a></div>
<div class="card"><h2>Red mug</h2>
<span class="stock">Sold out</span>
<a href="/p/2">View mug</a></div>
<div class="card"><h2>Green mug</h2>
<span class="price">$11</span>
<a href="/p/3">View mug</a></div>
```

```
extract=items:.card{title:h2|price:.price|url:a@href}*
```

```json
{"items": [
  {"title": "Blue mug", "price": "$9", "url": "https://lab.megalania.io/p/1"},
  {"title": "Red mug", "price": null, "url": "https://lab.megalania.io/p/2"},
  {"title": "Green mug", "price": "$11", "url": "https://lab.megalania.io/p/3"}
]}
```

Inside `{}` a field is a rule as above: `@attr`, `@json(path)` and a
trailing `*` work, and `id:@data-id` reads an attribute of the container
itself. Without the `*` after `}` you get the first container as one object,
or `null`. A row where every field is missing stays in the list, so the row
count is the container count. One level only: no `{}` inside `{}`. Fields
count toward the 32 rules. Row rules and plain rules mix in one string, at
the same price.

A name that matches nothing is `null`, and a list that matches nothing is `[]`,
so the shape of the object never changes between pages.

The answer is bounded, because a short selector can read a very large page:
8 MB of values in total, 1,000 matches for one list rule, and 1 MB for one
value. When the rules of one request together need too much selector work
on a very large page, the rules past that point are cut the same way, as
`null` or `[]`. So are the rules still running when the request's `timeout`
runs out: a slow extract stops there and returns what it has, instead of a
timeout. A `@json` rule reads a script when the request's scripts
together need at most 64 MiB of memory, counted as 5 bytes per byte of the
script plus 160 per value in it. A data script of a few megabytes fits; a
script past the bound is a miss, with `truncated`. A rule that reaches one of those stops there,
and the object gains one more key:

```json
{"title": "Widgets", "body": ["...", "..."], "truncated": true}
```

`truncated` means you got part of the answer, not an error: the page was
fetched, the request succeeded, and you are charged as usual. Narrow the
selector to get the rest. If the cut leaves every field empty and `json` is
the only output, the answer is a free 422 `convert_cap` on field `extract`. Because that key is ours, `truncated` is the one
name a rule cannot have; asking for it is a 400 naming `extract`.

`extract` and `response_types=json` go together. Either one without the other
is a 400 naming `extract`: a parameter that does nothing is a mistake we tell
you about rather than bill you for. A rule we cannot compile is also a 400
naming `extract`, before we fetch anything, so a typo costs nothing. The
message names the rule that is wrong and never repeats your selector back. `:has()` inside `:has()` is such a 400: CSS (Selectors Level 4) does not allow it, and browsers refuse it too. `:has()` matches like a browser: `li:has(li span)` needs the inner `li` and the `span` inside the outer `li`.

`markdown`, `text`, `links`, `metadata`, `page_data`, `compact_html` and `inspect` add nothing to the price: they are
the page you already paid for, in another shape. `json` is a priced feature of
its own. `X-Credits-Charged` on every response is what the request actually
cost.

## The identity, and staying one visitor

Every fetch looks like one real browser on the wire: its TLS hello, its
HTTP/2 frames and its headers agree, and they are checked against a real
browser's recording. There are two identities, `firefox` and `chrome`, both
on Windows. Leave `identity` out and we pick the one that does best on the
site; send `identity=firefox` or `identity=chrome` to choose. Both cost the
same, and `X-Identity` on every answer says which one fetched the page.

`session_id` keeps one visitor together: the same value leaves from the same
exit, keeps the identity that minted its cookies, and is never retried on
another exit. We keep the cookies a session collects on our side and send them
with the next request that names it, for at most 10 minutes, never to another
session or another identity. You never see them. A jar holds at most 50
cookies and 16 KB; a bigger one is dropped. You hold at most 100 jars. A new
`country_code`, `proxy_city` or `proxy_tier` starts a new, empty jar.
`session_id` pins the proxy tier: with no `proxy_tier` it is `residential`, so
adding `session_id` to a request that used to route to datacenter can raise
its price. Send `proxy_tier=datacenter` to keep the lower price.

The first hop always runs. Every later hop (a retry, a new exit, a higher
tier) starts only when the time left is at least what that path usually
needs. `X-Hops` lists the hops that ran and how long each took.

## Ask about a host before you spend credits

`GET /v1/stats/domain?host=<name>&days=7` answers how the whole platform does
on one host: how many requests, how many worked, how long the target took,
and how often it answered with a challenge. Same bearer token as `/v1`. `host`
follows the hostname rule in [Ask what a host costs](#ask-what-a-host-costs).

```sh
curl -s -H "Authorization: Bearer $MEG_API_KEY" \
  "https://get.megalania.io/v1/stats/domain?host=lab.megalania.io&days=7"
```

```json
{
  "host": "lab.megalania.io",
  "days": 7,
  "requests": 1420,
  "breakdown": [
    { "engine": "http", "tier": "residential",
      "requests": 1200, "success_rate": 0.972, "mean_target_ms": 840.5, "block_share": 0.021 },
    { "engine": "http", "tier": "datacenter",
      "requests": 220, "success_rate": 0.55, "mean_target_ms": 310.2, "block_share": 0.41 }
  ]
}
```

| parameter | values | default |
| -- | -- | -- |
| `host` | one hostname, `shop.example.com` | required |
| `days` | 1, 7 or 30 | 7 |

Read it like this: pick the cheapest line whose `success_rate` you can live
with. In the example above `datacenter` is the cheaper line and fails almost
half the time, so this host is worth `proxy_tier=residential`.

`success_rate` counts a page only when it came back with its content. An
empty JavaScript page (one that needs `engine=browser`) is not a success, even
when it was charged.

A line is `"engine": "http"` or `"engine": "browser"`. Leave `engine` and
`proxy_tier` out and we climb for you: from `datacenter` to `residential` on a
block, and to the browser when the page is a JavaScript shell.

Things to know:

- **The answer is per host.** `shop.example.com` and `example.com` are two
  answers: one site's hosts often sit behind different fronts. `www.` is
  dropped, so `www.example.com` and `example.com` are one answer. `host` in
  the body says which one you got.
- **The numbers are the whole platform's, not yours.** That is the point: a
  host you have never touched still tells you what to expect. Your own usage
  is a different endpoint.
- **The numbers update every hour.** They are rolled up at 20 minutes past
  each hour, so a fetch shows within about 80 minutes, today included.
  `days=1` is yesterday and today, `days=7` the last seven days and today.
- **Only fetches that reached the site count.** A request we refused before
  the fetch (balance, limits, capacity) says nothing about the site, so it is
  not in `requests` or `success_rate`.
- **A quiet line is left out.** A `breakdown` line counts only the days with
  at least 3 customers and at least 20 requests behind it, and `requests`
  counts only the lines shown. A customer here is an account that has paid or
  holds a beta invite; free sign-ups and test accounts do not count. With no line left we answer
  `"requests": 0` and an empty `breakdown`, the same answer a domain we have
  never seen gets. On a domain almost nobody scrapes, "the whole platform" is
  one customer, and their target is not ours to hand out.
- **Customers are counted per day.** For `days=7` and `days=30` a line needs 3
  customers on one day, so a line whose customers came on different days can
  stay hidden.
- **`days` is 1, 7 or 30.** Any other value is a 400 `bad_parameter` naming
  `days`.
- **A parameter we do not know is a 400**, the same as on `/v1`. This endpoint
  reads `host` and `days`. A
  misspelled `day=30` is `bad_parameter` naming `day`, rather than a quiet
  seven-day answer you would have read as the thirty you asked for.
- **One host per question.** There is no listing, no wildcard and no prefix.
  `host=*`, `host=.example.com` or a URL in `host` is a 400 `bad_parameter`
  naming `host`. Which domains we scrape is our customers' business, not a
  directory.
- **The answer is cached for five minutes.** Asking again inside that window
  costs you nothing and changes nothing.

## Ask what a host costs

`GET /v1/tier?host=<name>` answers the price tier of one host and what a
routed request to it costs you. Same bearer token as `/v1`. It is free and
writes no audit row.

`host` is one hostname, the same rule on `/v1/tier`, `/v1/stats/domain` and
the MCP tool `check_site`: letters, digits, hyphens and dots, each label 1 to
63 characters. No wildcard, underscore, IPv6 literal, scheme, port or path. A
trailing dot is dropped. Anything else is a 400 naming `host`.

```sh
curl -s -H "Authorization: Bearer $MEG_API_KEY" \
  "https://get.megalania.io/v1/tier?host=lab.megalania.io"
```

```json
{
  "host": "shop.example.com",
  "tier": 3,
  "temporary": false,
  "price": 16,
  "pending": { "tier": 5, "effective_on": "2026-09-29", "price": 60 },
  "quotes": { "datacenter": 1, "residential": 8, "browser": 15 },
  "capabilities": { "http": ["datacenter", "residential"], "browser": ["datacenter"] }
}
```

- `tier` is the tier in effect today, 1 to 5, and `price` is the quote for a
  typical page on this host in credits, from what fetching it cost us. It is
  not the charge: every request pays the price of the route that delivers the
  page, shown in `quotes`.
- `quotes` is what one request to this host costs you on each way we can
  fetch it today, in credits. With no `country_code` or `response_types`, each
  quote equals the base price of that route on the Plan page, on every host:
  the tier never changes what a route costs. Add `country_code` and `response_types`, read
  the same way `/v1` reads them, and the quotes include their price; without
  them, the quotes are before response types and premium geo. It has
  `datacenter` and `residential`, and `browser` only while a browser can
  serve you. `browser` is the browser on datacenter, and `browser_residential`
  the browser on residential IPs. Each is the price of a
  page up to the included bytes; a bigger page pays more for each started
  128 KiB, up to the byte cap. Your key's rungs decide
  which ways appear. `mobile` appears only while a mobile provider serves.
- `capabilities` lists, per engine, the proxy tiers that can serve you right
  now. An empty list means that engine cannot serve now. Read it before you
  set `engine` or `proxy_tier`.
- `temporary` is `true` while the host has too few requests for a measured
  tier. A temporary tier can rise at once.
- A host gets a new or updated tier only when at least 3 customers fetched
  it. A host that fewer than 3 customers fetched has no tier, the same as a
  new host. A tier it already has stays as it is.
- `pending` is a scheduled change and the day it takes effect. A rise of a
  measured tier comes 14 days after it is announced; a fall comes the next day.
- A host with no tier in effect yet answers `tier`, `temporary`, `price` and
  `pending` as `null`. No tier yet: the price depends on the route that answers.
  `quotes` shows that price for each route, the same on every site.
- `host` may not carry a tab, a newline or any other control character.
- `host` is one hostname: no scheme, port, path or wildcard, else 400
  `bad_parameter` naming `host`. `www.` and a trailing dot are dropped.
- It reads `host`, `country_code` and `response_types`. Any other key is a
  400 `bad_parameter` with no `field`.
- It shares your plan's concurrency limit with `/v1`: over it, 429
  `over_concurrency` with `Retry-After`.