Map a site

llms.txtUpdated 8 Nov 2026

Send a start URL, get the site's URLs back in one call. Then pick the pages you want and fetch them with POST /v1 or a batch. A map costs 4 credits on most sites. A map that finds no URL is free when the start page was blocked, failed or an error page (a 4xx or 5xx, such as a 404). A search that matches nothing on a page we read is still charged.

Shell

curl -s -X POST "https://get.megalania.io/v1/map" \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://lab.megalania.io/", "limit": 500}'

JSON

{
  "urls": [
    { "url": "https://lab.megalania.io/p/1", "source": "sitemap" },
    { "url": "https://lab.megalania.io/article", "source": "link" }
  ],
  "truncated": false
}

Try it on the Lab: a map of https://lab.megalania.io/ lists every Lab page from its sitemap, then the links on its home page.

What it reads

The start page, on the route our router picks for the site. The links on it that stay on the site become source: link.
/robots.txt, for the sitemaps it names. With none named, /sitemap.xml.
Those sitemaps, and the sitemaps a sitemap index lists, gzipped ones included: at most 10 files. Their URLs are source: sitemap.

That is at most 12 fetches. robots.txt and the sitemaps are text, so they use plain HTTP on the proxy tier the first fetch picked, never the browser. The list has sitemap URLs first, then start-page links, each URL once, without its #fragment. The map reads sitemaps only on the site’s own host (and its subdomains when you ask for them), never anywhere else. /robots.txt and the sitemap files are part of the map price and are never in the list.

The body

key	value	default
`url`	the page to map from, `http` or `https`. Its host is the site.	required
`limit`	the most URLs to return, 1 to 5000	500
`search`	keep only URLs that contain this text, 1 to 256 characters	none
`include_subdomains`	also keep URLs on subdomains of the site’s host	`false`

Any other key is a 400 bad_parameter naming it. Send an Idempotency-Key header to make a retry safe, as on /v1.

`truncated`

true means there may be more URLs than the list holds. Three things stop a map early: limit URLs were found, the route’s byte cap was reached, or your balance could not pay for more. A map with truncated: true still answers every URL it found. When a usage limit stopped it, such as today’s free browser and residential retries, the answer also carries "stopped": { "code": "free_failures_spent", "message": "..." }: the code is the 429 a new request would get.

What it costs

Map	Credits
HTTP datacenter, up to 384 KiB of wire	4
HTTP datacenter, above 384 KiB	4, plus 1 for each started 128 KiB above
On a residential or browser route	4, plus that route’s normal page and byte prices
No URL found, and the start page was blocked, failed or an error page (4xx, 5xx)	0

Every byte of every fetch counts, including fetches that failed and routes the router tried first. X-Credits-Charged says what the map cost. Most maps read well under 384 KiB.

Errors

The same codes as /v1; Error codes has them all. The ones you may see here: bad_parameter (400), bad_token (401), insufficient_credits (402: not enough for the cheapest route’s map), forbidden_target (403), replayed (409), body_too_large (413), a 429, and unavailable (503). None of them is charged.

Privacy

The URL list goes to you only. We do not store it and do not log it; the audit row keeps the host, as for any request.