Docs menu
- Docs
- Guides
- Fetch, extract and lists
Fetch, extract and lists
One endpoint, two ways in. POST /v1 with a JSON object is the main one. GET /v1?url=... is the paste-a-URL one. New here? Start with the quickstart. Error codes has every error.code, Batches sends many URLs in one call, Monitors watch pages on a schedule, Map a site lists a site's URLs in one call, and MCP for agents connects Claude, ChatGPT, Cursor or VS Code. Every parameter, its values and its default are in the API reference, generated from the API contract.
The scraping host is https://get.megalania.io.
Send your key (meg_sk_...) as Authorization: Bearer <your key>, on every route. That
is the only way in. A request with api_key in the query is a 401
bad_token, even when it also carries the header, and it is never charged.
A key in a URL is written down by everything the URL passes: your shell
history, your browser history, every proxy and CDN access log on the way, and
any Referer your page sends. We cannot clean those for you, so we do not
accept one. If a URL carrying your key was shared, revoke the key and make a new one.
Send Idempotency-Key: <your own id> on any call you might retry: /v1,
POST /v1/batches and checkout. It is 1 to 128 printable ASCII characters, and
we keep it for 24 hours. On /v1 a repeat is a 409 replayed that names the
first call’s X-Request-Id, never a second charge. On POST /v1/batches and
checkout a repeat returns the first answer. Error codes has the
rules.
QUERY /v1, the standards path
QUERY /v1 takes the same JSON envelope as POST /v1, on the same code path.
Everything is the same: the parameters, the errors, Idempotency-Key, the
price. Only the method word changes.
It exists because QUERY is the right method for this call. RFC 10008
(Proposed Standard, June 2026) added it for a request that carries a body and
is safe and idempotent — which a render is: it reads a page and changes nothing
at the target.
POST stays the example here. Browsers are still evaluating QUERY, and
an older WAF, corporate proxy or load balancer in front of you may refuse a
method it does not know, usually as a 405 or a 501 that never reaches us. Try
QUERY from a server-side client, keep POST as the fallback, and use POST
wherever a browser or someone else’s proxy is in the path.
curl --request QUERY \
--header "Authorization: Bearer $MEG_API_KEY" \
--header "Content-Type: application/json" \
--data '{"url": "https://docs.megalania.io/start/quickstart"}' \
"https://get.megalania.io/v1"Every answer from /v1 carries Cache-Control: no-store, on all three
methods. A render is your live data and must never be handed to you from a
cache your side keeps, such as a proxy of yours. This matters most on QUERY,
whose answers a cache may keep by default, but it is on every answer so no
method is the exception. Our own short page cache is described in
Reuse a recent copy.
The JSON envelope
curl --header "Authorization: Bearer $MEG_API_KEY" \
--header "Content-Type: application/json" \
--data '{
"url": "https://example.com/search?q=shoes&sort=price",
"country_code": "US",
"method": "POST",
"content_type": "application/x-www-form-urlencoded",
"body": "cT1zaG9lcw=="
}' \
"https://get.megalania.io/v1"One JSON object. Its keys are the parameter names of the GET, plus three
that exist only here:
| key | values | default |
|---|---|---|
method |
GET or POST |
GET |
body |
the target’s own request body, base64 | none |
content_type |
the media type of body, for example application/json |
none |
body and content_type need method: POST; on a GET they are a 400, so a
body is never dropped silently after you were charged.
Every value is a string, except only_main_content, which also takes JSON
true or false; timeout, max_credits and max_age, which also take a
whole JSON number; and headers, a list of strings, each name:value. Any other type is a 400 naming the key.
Nothing of ours is in the query string on this route, so a target URL keeps
its own query string with no encoding at all, and an unknown key is the same
400 it is on the GET. The envelope is at most 1 MB; over that is a 413.
There is no query string on this route. POST /v1?url=... is a 400, so a
parameter you left in the URL is never quietly dropped after you were charged.
The token is the Authorization header here too: we verify it before we read
a single byte of the body.
Leave engine and proxy_tier out and we pick the cheapest way to fetch the
page for you: plain HTTP on a datacenter exit first, a residential exit when
the site blocks us. You pay the price of the route that delivers the page, and
only when the page comes back. A page that builds itself in JavaScript needs
engine=browser, which sends it to a real browser (Chrome, datacenter by
default): 15 credits and up. wait_until other than load does the
same. Add proxy_tier=residential to run the browser on residential IPs (32
credits and up). Engines and proxy tiers explains the climb.
The browser takes GET only, without session_id or identity. Try it on the
Lab: https://lab.megalania.io/js?seed=7 is empty over HTTP and shows
“seed 7” with engine=browser.
max_credits (1 to 32767) is the most one page may cost. We skip any route
above it. If no route is left, the answer is a free 400
max_credits_too_low, before any attempt. A page whose bytes cost more is
cut at the bytes max_credits pays for: a 200 with X-Truncated: max_credits.
screenshot=viewport or screenshot=full adds a JPEG of the page as the
browser rendered it: the visible window, or the whole page up to 16,384
pixels tall. It needs the browser, so it sends the request there;
engine=http with screenshot is a free 400. The answer becomes
multipart/mixed with one more part, screenshot:
{"url": "...", "expires_at": "...", "bytes": n, "credits": 1}. The link
works for 1 hour, for anyone who holds it, and the image is deleted after 7
days. X-Credits-Charged is the page alone; the part’s credits is what the
screenshot adds: 1 for viewport, 2 for full, charged only once the image
is stored and its link is in the part. If the screenshot could not be made or
stored, the part says {"error": "screenshot_failed", "credits": 0} and
nothing is charged for it. A blocked page is free, screenshot included.
Batches take screenshot too: the item’s result holds the screenshot part
with bytes and credits, and each read of the batch’s items signs a fresh
screenshot_url that works for 1 hour.
Page actions
actions runs steps in the browser after the page loads and wait_until is
met, in order. Then we read the page and take any screenshot. Use it to click
“load more”, fill a search box, or scroll a list.
actions=click:.more|wait_for:.item-2|type:#q=shoes|press:Enter|wait_for:.results
Entries are kind:argument, joined by |. At most 20:
| entry | what it does |
|---|---|
wait:<ms> |
waits 1 to 10,000 ms |
wait_for:<css> |
waits up to 10 s for a visible element that matches |
click:<css> |
clicks the first visible match, with a real mouse event; waits up to 5 s for it |
type:<css>=<text> |
focuses the first visible match and types the text, 1 to 1,024 bytes |
press:<key> |
one key: Enter, Tab, Escape, Backspace, Space, ArrowUp, ArrowDown, ArrowLeft, ArrowRight, PageUp, PageDown, Home or End |
scroll:<where> |
bottom, top, or a number of pixels down |
Try it on the Lab: https://lab.megalania.io/scroll sends ten items, and each
scroll to the bottom adds ten more.
actions=scroll:bottom|wait_for:#item-20|scroll:bottom|wait_for:#item-30
- All actions together get at most 30 s, inside
timeout. A click that opens a new page is followed: the status and the HTML are the new page’s. - A selector is at most 1,024 bytes.
|ends an entry, so a selector or a text cannot hold it ([lang|=en]cannot be sent). Selectors do not reach into iframes or shadow roots. - Send
typebyPOST /v1, so the text is not in a URL. actionsneeds the browser, so it sends the request there;engine=httpwithactionsis a free 400 namingactions.- A page with actions costs 3 credits more, charged once with the page. See Credits and pricing.
- If an action cannot run (no visible match in time, or the 30 s ran out),
the answer is a free 422
action_failed.error.fieldisactionsand the message names the entry’s index (0 is the first). It never repeats your selector or text, and it carries no page, cookies or target status. - If every action ran and the page then blocks us, the answer is a free 422
blocked, as always. If an action failed, the answer isaction_failed, even on a blocked page. The failure cap charges failures above 30% an hour. - Batches take
actionstoo, with the same prices and the same failure cap.
Network capture
network=true also returns the JSON and text the page itself loaded with
XHR or fetch: an app’s own API answers, often cleaner than the HTML. It
runs from the page load through any actions, until we read the page.
GET /v1?url=https://shop.example.com/list&network=true&network_url_globs=*/api/*
The answer is multipart/mixed with one more part, network, in
application/x-ndjson: one JSON object per response, one per line.
{"url":"https://shop.example.com/api/items?page=1","method":"GET","status":200,"content_type":"application/json","bytes":5120,"truncated":false,"body":"{\"items\":[...]}"}bodyis UTF-8 text. A body that is not text is base64, with"encoding": "base64".bytesis the body’s size. A body over 2 MiB is not kept:bodyis empty,truncatedistrue, andbytesis its encoded size on the wire (compressed, headers included).- Without
network_url_globswe capture JSON, NDJSON and text responses.network_url_globskeeps only the responses whose full URL matches one of its patterns, of any content type: a comma list of at most 10, 1,024 bytes in all, where*matches anything and?one character. It needsnetwork=true. - Caps: at most 200 responses, 1 MiB per body (then
truncatedistrue), 8 MiB in all, and URLs up to 8 KiB. When a cap cut the list, the answer hasX-Network-Truncated: true. It is also set when a response we would keep was still loading when we read the page. A request that never got an answer, or a body type we would not keep, does not set it. - Control characters in captured bodies are JSON-escaped and count toward the response size.
- Only the page’s own XHR and
fetchare captured. Requests from web workers and service workers are not. - We return no request or response headers, no cookies and no request bodies. Captured URLs can carry the page’s own tokens in their query strings: they are yours, and we never log or store them.
- It adds no credits: the page’s bytes are already metered. See Credits and pricing.
networkneeds the browser, so it sends the request there;engine=httpwithnetwork=trueis a free 400 namingnetwork.networkwithschemais a free 400 namingnetwork.- A blocked page is free, and carries no network part.
- Batches take
networktoo: the item’s result holds the network part.
Some fields are not supported yet, and each answer is a free 400
bad_parameter:
networkinresponse_types: the error namesresponse_types. Sendnetwork=trueinstead.steps=as a key of its own: the error says it is not a parameter.
country_code defaults to
whatever the deployment is configured with, US on the dev cluster.
Countries lists what each route serves, and GET /v1/geos
gives it live. A forced route that does not serve the country answers a free
400 on country_code. X-Geo says which country the request exited from.
What you get back
response_types decides the shape. One type is the body itself, with its own
content type. Several is multipart/mixed, one part per type, in the order you
asked for them.
| type | content type | what it is | credits |
|---|---|---|---|
html |
text/html; charset=utf-8 |
the page, the default | in base |
markdown |
text/markdown; charset=utf-8 |
the same page as Markdown | 0 |
text |
text/plain; charset=utf-8 |
the same content as plain text | 0 |
links |
application/json |
every http(s) link on the page, with its anchor text |
0 |
metadata |
application/json |
title, description, language, canonical, OpenGraph and JSON-LD | 0 |
page_data |
application/json |
the data the page embeds: JSON-LD and named JSON state such as __NEXT_DATA__ |
0 |
compact_html |
text/html; charset=utf-8 |
the page’s HTML without scripts, styles and noise; CSS selectors still match | 0 |
inspect |
application/json |
the page’s structure: repeated blocks with ready extract rules, headings, links, forms and embedded data |
0 |
json |
application/json |
the fields your extract rules read |
1 |
The credits column is the dev prices, and every number in it is a row of the
billing_price table rather than anything built into the service. Ask for
several types and you pay for each of them on top of the base request.
Documents: PDF, Word, PowerPoint and Excel
When the url answers with a PDF, DOCX, PPTX or XLSX file, you get the same
response_types as for a page. We read the file’s own bytes to know what it
is, not its name or its Content-Type. markdown keeps headings, lists,
tables and links. text, links, metadata, json with extract rules, and
schema all work on it. html is the file turned into HTML, not the raw
bytes. Batches and the MCP scrape tool take documents the same way.
| file | priced per | dev credits |
|---|---|---|
| page | 1 | |
| PPTX | slide | 1 |
| DOCX | 3,000 characters of text, rounded up | 1 |
| XLSX | 1,000 non-empty cells, rounded up | 1 |
This is on top of the base request. We count the pages after the file
arrives. If your balance cannot cover them, you get a 402
insufficient_credits; if your max_credits cannot, a 400
max_credits_too_low. Either way nothing is sent and nothing
is charged, and the request counts as a free failure.
Limits: 500 pages (or 500 slides, 1,500,000 characters, 500,000 cells), 20,000
characters of text per page or slide, and 64 MiB once the file is unpacked.
Past the page or text limit we convert the first part and say so in
X-Document-Truncated: true. Past the unpacked size it is a free 422
convert_cap. We read each file in a separate process with its own memory and
time limit. A scanned PDF with no text layer, a file that needs a password, a
damaged file, or one that runs past those limits is a free 422
document_unreadable. We do not run OCR yet. A screenshot of a document is not
charged.
The answer carries X-Document: pdf (or docx, pptx, xlsx) and
X-Document-Units: the pages, slides, characters or cells we counted. Document
content is never logged.
Structured extraction by schema
Send response_types=json with a schema instead of extract, and a
language model fills your JSON Schema from the page’s main content:
curl -s -H "Authorization: Bearer $MEG_API_KEY" -G "https://get.megalania.io/v1" \
--data-urlencode "url=https://lab.megalania.io/p/1" \
--data-urlencode "response_types=json" \
--data-urlencode 'schema={"type":"object","additionalProperties":false,"required":["title"],"properties":{"title":{"type":"string"},"price":{"type":"number"}}}'{"data": {"title": "Blue mug", "price": 9.0}, "credits": 26, "tokens": 7200}schemais one JSON Schema object of at most 8 KB, withtype: objectat the top andadditionalProperties: falseon every object. It may usetype,properties,required,items,enum,const,anyOf,format,descriptionandtitle.prompt(at most 2 KB) adds your instructions, for example “the main product, not the recommendations”.extract_modelisfast,defaultornon_china, never a vendor name. We never use a model of China origin, whatever its host. Today all three aliases run claude-haiku-4-5.X-Credits-Chargedandcreditsin the answer are the same number: the whole request, the page plus the extraction.llm_creditsis the extraction’s part: 2, plus a rate per started 1,000 weighted tokens, where weighted tokens are input + 5 × output (tokens). Credits and pricing says where the rates live.- An answer that does not match the schema is a
422 bad_extract. The extraction is free, the page is still charged, and the body carries the page as markdown so you can see what the model saw. After 20 of them on your account in a UTC day, the next extraction is a free429 llm_token_capuntil 00:00 UTC. - If your balance covers the page but not the extraction, the answer is a
402 insufficient_credits: the page is charged, the extraction is free, and the body carries the page as markdown. screenshotdoes not go withschema: that is a free 400 namingscreenshot.- Limits: 96 KB of markdown per request (a longer page is cut, with a
marker), 8,192 output tokens, and a daily allowance of weighted tokens per
key and per account (1M in the beta). Over it, the answer is a free
429 llm_token_cap. - Before the model runs, we reserve the most it can cost: one input token
per byte we send, plus 8,192 output tokens. You are charged what the model
really used, never more than that reserve. With
max_credits, the reserve must fit under it, or the extraction is skipped, free. - Your page content leaves us. The page’s main content, your schema and
your prompt go to the model provider: Anthropic in the US, never with your account,
key or who you are. Anthropic may keep it for a time under its own terms. This
happens only when you send
schema. See Privacy. - Batches and crawls take
schematoo: each item’s result is the JSON answer, charged the same way as one page.
Reading several types at once
Parts come in the order you listed them in response_types. The boundary is
in the Content-Type header of the response. Each part has its own
Content-Type and a Content-Disposition: form-data; name="<type>", where
<type> is the type’s lowercase name. A type with nothing in it,
such as links on a page with no links, still gets its part, with an empty
body. Save the headers and the body:
curl -sS -D head.txt -o body.bin -H "Authorization: Bearer $MEG_API_KEY" \
"https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown,links,metadata"Then split the body with the Python standard library:
import json
import re
head = open("head.txt", encoding="latin-1").read()
ctype = [l.split(":", 1)[1].strip() for l in head.splitlines()
if l.lower().startswith("content-type:")][-1]
boundary = ("--" + ctype.split("boundary=", 1)[1]).encode()
body = open("body.bin", "rb").read()
assert body.endswith(boundary + b"--\r\n"), "cut short: do not trust the last part"
parts = {}
for chunk in body.split(boundary)[1:-1]:
part_head, _, data = chunk.partition(b"\r\n\r\n")
name = re.search(rb'name="([a-z_]+)"', part_head).group(1).decode()
parts[name] = data[:-2]
markdown = parts["markdown"].decode()
links = json.loads(parts["links"] or b"[]")
metadata = json.loads(parts["metadata"] or b"{}")
print(sorted(parts), len(links), metadata.get("title"))One type is never multipart: the body is the part itself.
markdown is the page converted for you: headings, links, images, lists,
tables, code. script, style and hidden elements are always dropped. With
only_main_content (the default), nav, aside, form controls and a page
header or footer are dropped too; a header or footer inside the article
is kept. With only_main_content=false they all stay.
On a product page, markdown keeps what a shopper needs. A price shown only
to sighted users (the visible text is aria-hidden) comes from the label of
its group, once. Option buttons become one line, for example
Size: XS, S, M (sold out). A block the page repeats for mobile and desktop
shows once. Collapsed panels (<details>, an accordion a button opens) are
kept. When the page has exactly one JSON-LD Product, the markdown ends with
Product data from the page (JSON-LD): and its name, brand, price, availability,
rating, SKU and variant count. compact_html keeps the same price, options
and collapsed panels; it has no JSON-LD block.
markdown, text and compact_html are the target page’s own content. If
you render them as HTML, sanitize them first: they can contain links and markup
the page author chose, and compact_html keeps forms, id, name and remote
images. For an LLM, treat page text as untrusted input: a page can hide
instructions in text that users do not see. For price monitoring, use
markdown or metadata: text output has no strike marker, so an old price and a
new price look alike.
A page that shows its content only after JavaScript runs is rendered in a real
browser. Leave engine out and the ladder moves to the browser on its own when
your plan and max_credits allow it. When it cannot, the empty shell comes back
free, marked X-Hint: needs_js; send engine=browser to render it.
Ask for several types in one request, and they come from one read of the page. You pay for one fetch:
curl -H "Authorization: Bearer $MEG_API_KEY" \
"https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown,text,links,metadata"textis themarkdowncontent with no markup. Paragraphs stay apart.linksis everyhttp(s)link, absolute, once each, with its anchor text.metadatais the requested URL, title, description, language, canonical, the OpenGraph fields and every JSON-LD object on the page, as it was. When the page names its next page (rel="next", or one “next” link in a pagination block),next_pageholds that link, absolute, when it stays on the page’s own site. We never build it from a pattern. On a product page, the JSON-LD often holds the price and the specs more reliably than the page text.page_datais the data the page carries for its own scripts:{"json_ld": [...], "state": {"<script id>": ...}}.json_ldis every JSON-LD object;stateis every<script type="application/json">with anid, such as__NEXT_DATA__or__NUXT_DATA__, parsed. A listing page often holds every item there, and only some of them in the visible HTML. Ask formarkdown,page_datato get both from one fetch. It is untrusted page content, like the rest.compact_htmlis the page’s HTML made small for an LLM or a selector test. We remove scripts, styles, comments,svg,canvas,iframe,linkandmeta, hidden elements (hidden,aria-hidden="true", inlinedisplay:none), empty elements and every attribute exceptid,class,href,src,alt,title,name,type,value,datetime,lang,role,aria-label,itemprop,itemtype,data-testidanddata-test(andcontentnext toitemprop).hrefandsrcare made absolute; ajavascript:,data:or other non-http link is removed. Whitespace is collapsed, except insidepreandtextarea. Thetitlestays first.idandclassstay, so a selector you test on the raw page still matches. It is cut at about 512 KiB, at an element boundary with every tag closed, and then the answer hasX-Compact-Truncated: true. Likemarkdown, it is the main content unless you sendonly_main_content=false.inspectis the page’s structure, to help you writeextractrules without reading the page:{"title", "containers", "headings", "links", "forms", "data"}. Each container is a block that repeats (a product card, a search result) with itsselector, itscount, a shortsample, a readyextractvalue to send as theextractparameter, and theextractof each field inside it.datalists the JSON scripts the page embeds, their key paths and anextractthat reads them with@json. It never holds the page’s text beyond short samples. It reads the whole page, soonly_main_contentdoes not change it. The MCPinspecttool returns the same object. Wheninspectis the only output and the page shows no structure, the answer is a free422 convert_capon fieldengine: the page may build its content with JavaScript, so sendengine=browser. When the site answers 4xx,inspectanswers200withX-Target-Status, the structure as found and anotesuch as “The site answered 404, so there is little page structure to inspect.”, charged like any answer under 500.
markdown and text hold the page’s main content by default. Some pages keep
a price or a spec table outside it. Send only_main_content=false to get the
whole page instead:
curl -H "Authorization: Bearer $MEG_API_KEY" \
"https://get.megalania.io/v1?url=https://lab.megalania.io/article&response_types=markdown&only_main_content=false"only_main_content is true or false in any case, and true is the
default. Anything else is a 400 naming only_main_content. false changes only markdown,
text and compact_html, so it needs one of them in response_types; without them it is also a
400.
markdown, text, links, metadata, page_data, compact_html, inspect and json have size limits:
- The page we convert can be at most 8 MiB of HTML.
- The converted output has a budget. The budget grows with the size of the page, so a normal page never comes near it.
- A page over either limit, or one our HTML parser cannot read (for example,
nested too deep), is a 422
convert_capwitherror.fieldresponse_types. You are not charged. - The converted types of one request share one budget. Asking for
compact_htmlnext tomarkdowncan turn a very large page into a freeconvert_capwheremarkdownalone would have fit. - To fix it, ask for fewer
response_types, or ask forhtmland convert the page yourself.htmlhas no such limit.
json reads fields out of the page with your own selectors, in extract:
extract=title:h1|price:.price|links:a@href*
One rule is <name>:<css>. At most 32 rules and 4096 bytes, joined by |. Add
@attr to read an attribute instead of the text, and a trailing * to get
every match as a list instead of the first one. That example answers:
{"title": "Widgets", "price": "$9.99", "links": ["/a", "/b"]}To read a script’s JSON (for example __NEXT_DATA__) without the raw page, end the
rule with @json(path): the path is dot-separated keys and array indexes, and
the value comes back as JSON, not text. With *, an array gives its elements.
Same price as any json rule.
extract=products:script#__NEXT_DATA__@json(props.pageProps.products)*
A script that is not JSON, or a path that is not there, is a miss.
When each element is a large object, name only the parts you need in {}
after the path. Each field is a dot path inside the element; a key that is
not there is null, so every row has the same keys. !key drops the
elements whose key is true (also a non-empty string, a non-zero number, or a
non-empty list or object); it is the only filter. Fields and !key count
toward the 32 rules.
extract=items:script#__NEXT_DATA__@json(props.pageProps.items)*{!isSponsored|name:name|price:price.current|link:url}{"items": [{"name": "Oak lamp", "price": "$29.00", "link": "/p/1"}]}For a list of cards, wrap the fields in {} after the container selector.
Each match of the container gives one object, and the fields look only inside
it. A field that is missing in one card is null in that card, so rows never
shift. On the Lab’s cards page, https://lab.megalania.io/cards, the second
card has no price:
<div class="card"><h2>Blue mug</h2> <span class="price">$9</span> <a href="/p/1">View mug</a></div> <div class="card"><h2>Red mug</h2> <span class="stock">Sold out</span> <a href="/p/2">View mug</a></div> <div class="card"><h2>Green mug</h2> <span class="price">$11</span> <a href="/p/3">View mug</a></div>
extract=items:.card{title:h2|price:.price|url:a@href}*{"items": [
{"title": "Blue mug", "price": "$9", "url": "https://lab.megalania.io/p/1"},
{"title": "Red mug", "price": null, "url": "https://lab.megalania.io/p/2"},
{"title": "Green mug", "price": "$11", "url": "https://lab.megalania.io/p/3"}
]}Inside {} a field is a rule as above: @attr, @json(path) and a
trailing * work, and id:@data-id reads an attribute of the container
itself. Without the * after } you get the first container as one object,
or null. A row where every field is missing stays in the list, so the row
count is the container count. One level only: no {} inside {}. Fields
count toward the 32 rules. Row rules and plain rules mix in one string, at
the same price.
A name that matches nothing is null, and a list that matches nothing is [],
so the shape of the object never changes between pages.
The answer is bounded, because a short selector can read a very large page:
8 MB of values in total, 1,000 matches for one list rule, and 1 MB for one
value. When the rules of one request together need too much selector work
on a very large page, the rules past that point are cut the same way, as
null or []. So are the rules still running when the request’s timeout
runs out: a slow extract stops there and returns what it has, instead of a
timeout. A @json rule reads a script when the request’s scripts
together need at most 64 MiB of memory, counted as 5 bytes per byte of the
script plus 160 per value in it. A data script of a few megabytes fits; a
script past the bound is a miss, with truncated. A rule that reaches one of those stops there,
and the object gains one more key:
{"title": "Widgets", "body": ["...", "..."], "truncated": true}truncated means you got part of the answer, not an error: the page was
fetched, the request succeeded, and you are charged as usual. Narrow the
selector to get the rest. If the cut leaves every field empty and json is
the only output, the answer is a free 422 convert_cap on field extract. Because that key is ours, truncated is the one
name a rule cannot have; asking for it is a 400 naming extract.
extract and response_types=json go together. Either one without the other
is a 400 naming extract: a parameter that does nothing is a mistake we tell
you about rather than bill you for. A rule we cannot compile is also a 400
naming extract, before we fetch anything, so a typo costs nothing. The
message names the rule that is wrong and never repeats your selector back. :has() inside :has() is such a 400: CSS (Selectors Level 4) does not allow it, and browsers refuse it too. :has() matches like a browser: li:has(li span) needs the inner li and the span inside the outer li.
markdown, text, links, metadata, page_data, compact_html and inspect add nothing to the price: they are
the page you already paid for, in another shape. json is a priced feature of
its own. X-Credits-Charged on every response is what the request actually
cost.
The identity, and staying one visitor
Every fetch looks like one real browser on the wire: its TLS hello, its
HTTP/2 frames and its headers agree, and they are checked against a real
browser’s recording. There are two identities, firefox and chrome, both
on Windows. Leave identity out and we pick the one that does best on the
site; send identity=firefox or identity=chrome to choose. Both cost the
same, and X-Identity on every answer says which one fetched the page.
session_id keeps one visitor together: the same value leaves from the same
exit, keeps the identity that minted its cookies, and is never retried on
another exit. We keep the cookies a session collects on our side and send them
with the next request that names it, for at most 10 minutes, never to another
session or another identity. You never see them. A jar holds at most 50
cookies and 16 KB; a bigger one is dropped. You hold at most 100 jars. A new
country_code, proxy_city or proxy_tier starts a new, empty jar.
session_id pins the proxy tier: with no proxy_tier it is residential, so
adding session_id to a request that used to route to datacenter can raise
its price. Send proxy_tier=datacenter to keep the lower price.
The first hop always runs. Every later hop (a retry, a new exit, a higher
tier) starts only when the time left is at least what that path usually
needs. X-Hops lists the hops that ran and how long each took.
Ask about a host before you spend credits
GET /v1/stats/domain?host=<name>&days=7 answers how the whole platform does
on one host: how many requests, how many worked, how long the target took,
and how often it answered with a challenge. Same bearer token as /v1. host
follows the hostname rule in Ask what a host costs.
curl -s -H "Authorization: Bearer $MEG_API_KEY" \ "https://get.megalania.io/v1/stats/domain?host=lab.megalania.io&days=7"
{
"host": "lab.megalania.io",
"days": 7,
"requests": 1420,
"breakdown": [
{ "engine": "http", "tier": "residential",
"requests": 1200, "success_rate": 0.972, "mean_target_ms": 840.5, "block_share": 0.021 },
{ "engine": "http", "tier": "datacenter",
"requests": 220, "success_rate": 0.55, "mean_target_ms": 310.2, "block_share": 0.41 }
]
}| parameter | values | default |
|---|---|---|
host |
one hostname, shop.example.com |
required |
days |
1, 7 or 30 | 7 |
Read it like this: pick the cheapest line whose success_rate you can live
with. In the example above datacenter is the cheaper line and fails almost
half the time, so this host is worth proxy_tier=residential.
success_rate counts a page only when it came back with its content. An
empty JavaScript page (one that needs engine=browser) is not a success, even
when it was charged.
A line is "engine": "http" or "engine": "browser". Leave engine and
proxy_tier out and we climb for you: from datacenter to residential on a
block, and to the browser when the page is a JavaScript shell.
Things to know:
- The answer is per host.
shop.example.comandexample.comare two answers: one site’s hosts often sit behind different fronts.www.is dropped, sowww.example.comandexample.comare one answer.hostin the body says which one you got. - The numbers are the whole platform’s, not yours. That is the point: a host you have never touched still tells you what to expect. Your own usage is a different endpoint.
- The numbers update every hour. They are rolled up at 20 minutes past
each hour, so a fetch shows within about 80 minutes, today included.
days=1is yesterday and today,days=7the last seven days and today. - Only fetches that reached the site count. A request we refused before
the fetch (balance, limits, capacity) says nothing about the site, so it is
not in
requestsorsuccess_rate. - A quiet line is left out. A
breakdownline counts only the days with at least 3 customers and at least 20 requests behind it, andrequestscounts only the lines shown. A customer here is an account that has paid or holds a beta invite; free sign-ups and test accounts do not count. With no line left we answer"requests": 0and an emptybreakdown, the same answer a domain we have never seen gets. On a domain almost nobody scrapes, “the whole platform” is one customer, and their target is not ours to hand out. - Customers are counted per day. For
days=7anddays=30a line needs 3 customers on one day, so a line whose customers came on different days can stay hidden. daysis 1, 7 or 30. Any other value is a 400bad_parameternamingdays.- A parameter we do not know is a 400, the same as on
/v1. This endpoint readshostanddays. A misspelledday=30isbad_parameternamingday, rather than a quiet seven-day answer you would have read as the thirty you asked for. - One host per question. There is no listing, no wildcard and no prefix.
host=*,host=.example.comor a URL inhostis a 400bad_parameternaminghost. Which domains we scrape is our customers’ business, not a directory. - The answer is cached for five minutes. Asking again inside that window costs you nothing and changes nothing.
Ask what a host costs
GET /v1/tier?host=<name> answers the price tier of one host and what a
routed request to it costs you. Same bearer token as /v1. It is free and
writes no audit row.
host is one hostname, the same rule on /v1/tier, /v1/stats/domain and
the MCP tool check_site: letters, digits, hyphens and dots, each label 1 to
63 characters. No wildcard, underscore, IPv6 literal, scheme, port or path. A
trailing dot is dropped. Anything else is a 400 naming host.
curl -s -H "Authorization: Bearer $MEG_API_KEY" \ "https://get.megalania.io/v1/tier?host=lab.megalania.io"
{
"host": "shop.example.com",
"tier": 3,
"temporary": false,
"price": 16,
"pending": { "tier": 5, "effective_on": "2026-09-29", "price": 60 },
"quotes": { "datacenter": 1, "residential": 8, "browser": 15 },
"capabilities": { "http": ["datacenter", "residential"], "browser": ["datacenter"] }
}tieris the tier in effect today, 1 to 5, andpriceis the quote for a typical page on this host in credits, from what fetching it cost us. It is not the charge: every request pays the price of the route that delivers the page, shown inquotes.quotesis what one request to this host costs you on each way we can fetch it today, in credits. With nocountry_codeorresponse_types, each quote equals the base price of that route on the Plan page, on every host: the tier never changes what a route costs. Addcountry_codeandresponse_types, read the same way/v1reads them, and the quotes include their price; without them, the quotes are before response types and premium geo. It hasdatacenterandresidential, andbrowseronly while a browser can serve you.browseris the browser on datacenter, andbrowser_residentialthe browser on residential IPs. Each is the price of a page up to the included bytes; a bigger page pays more for each started 128 KiB, up to the byte cap. Your key’s rungs decide which ways appear.mobileappears only while a mobile provider serves.capabilitieslists, per engine, the proxy tiers that can serve you right now. An empty list means that engine cannot serve now. Read it before you setengineorproxy_tier.temporaryistruewhile the host has too few requests for a measured tier. A temporary tier can rise at once.- A host gets a new or updated tier only when at least 3 customers fetched it. A host that fewer than 3 customers fetched has no tier, the same as a new host. A tier it already has stays as it is.
pendingis a scheduled change and the day it takes effect. A rise of a measured tier comes 14 days after it is announced; a fall comes the next day.- A host with no tier in effect yet answers
tier,temporary,priceandpendingasnull. No tier yet: the price depends on the route that answers.quotesshows that price for each route, the same on every site. hostmay not carry a tab, a newline or any other control character.hostis one hostname: no scheme, port, path or wildcard, else 400bad_parameternaminghost.www.and a trailing dot are dropped.- It reads
host,country_codeandresponse_types. Any other key is a 400bad_parameterwith nofield. - It shares your plan’s concurrency limit with
/v1: over it, 429over_concurrencywithRetry-After.