TypeScript SDK

install megalania, then new Megalania().scrape({ url }) (the key comes from MEG_API_KEY on Node). You get the Response, the headers as typed fields, and a MegalaniaError with the error code and message. It retries once on a 429, after Retry-After, and never on anything else. Every scrape, map and create call carries an Idempotency-Key, so a repeat is never charged twice. No runtime dependencies: it uses the built-in fetch.

The SDK is in beta. Its types are generated from the OpenAPI document (api/openapi/v1.yaml), so they cannot drift from the API. The code is MIT licensed at https://github.com/megalania/megalania-typescript. The Vercel AI SDK tools build on it.

Install

Node.js 22.18 or newer, Deno or Bun.

npm install megalania

Scrape, extract and batch the Lab

Three first calls against the Lab, our test site: one page, rows from a list page, and a batch of both. Each one runs as pasted once MEG_API_KEY is set. The dashboard’s SDKs & code page shows the same code.

One page as Markdown

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const page = await meg.scrape({ url: "https://lab.megalania.io/article", response_types: "markdown" });
console.log(page.headers["X-Credits-Charged"], (await page.response.text()).slice(0, 300));

Rows from a list page

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const rows = await meg.scrape({ url: "https://lab.megalania.io/cards", response_types: "json", extract: "items:.card{title:h2|price:.price|url:a@href}*" });
console.log(rows.headers["X-Credits-Charged"], (await rows.response.text()).slice(0, 300));

Many pages in one batch

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const batch = await meg.createBatch({ urls: ["https://lab.megalania.io/article", "https://lab.megalania.io/cards"], response_types: "markdown" });
console.log(batch.id, (await meg.getBatch(String(batch.id))).status);

One page

Save as scrape.mjs and run node scrape.mjs:

import { Megalania, MegalaniaError } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, {
  baseUrl: process.env.MEG_BASE_URL ?? "https://get.megalania.io",
});
try {
  const { headers, response } = await meg.scrape({ url: "https://example.com/", country_code: "US" });
  console.log(headers["X-Credits-Charged"], headers["X-Target-Status"], headers["X-Request-Id"]);
  console.log((await response.text()).slice(0, 200));
} catch (e) {
  if (e instanceof MegalaniaError) console.log(e.status, e.code, e.field, e.message, e.requestId);
  throw e;
}

scrape(envelope) takes the JSON envelope as an object, typed as ScrapeEnvelope. It sends only the keys you set. A key you leave out gets the server’s default, which is not always the same as sending that default yourself: a proxy_tier you leave out lets us pick the cheapest route that works.

scrape sends an Idempotency-Key: a fresh UUID, or the { idempotencyKey: "..." } you pass as the second argument. Pass your own on a call you might send again yourself. A repeat is a 409 replayed, never a second charge.

Batches, monitors, map and tier

Batches and monitors live on the API host, https://api.megalania.io. Pass { apiUrl } to change it. Each method is named after its REST operation and resolves to the JSON answer.

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
console.log(await meg.getTier("docs.megalania.io"));
console.log(await meg.mapSite({ url: "https://docs.megalania.io/start/quickstart", limit: 20 }));
const batch = await meg.createBatch({ urls: ["https://docs.megalania.io/start/quickstart"], response_types: "markdown" });
console.log(await meg.getBatch(batch.id));
console.log(await meg.listBatchItems(batch.id));
const monitor = await meg.createMonitor({ name: "Demo", urls: ["https://docs.megalania.io/start/quickstart"], every: "1d" });
await meg.updateMonitor(monitor.id, { paused: true });

scrape, mapSite, createBatch and createMonitor send an Idempotency-Key, yours or a fresh one. The 429 rule and MegalaniaError are the same as for scrape.

Read many pages

waitBatch(id) polls until a batch is done, failed or canceled. batchResults(id) then yields each finished item with its body. It fetches several result links at once, and it never sends your key to them. readPages(urls) scrapes one page at a time, or crawls from one URL with { crawl }. It yields [text, metadata]. With a schema, it asks for json, and the text is the filled schema.

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
const batch = await meg.createBatch({ urls: ["https://docs.megalania.io/start/quickstart"], response_types: "markdown" });
console.log((await meg.waitBatch(String(batch.id))).status);
for await (const [item, body] of meg.batchResults(String(batch.id))) console.log(item.url, body.length);
const schema = { type: "object", additionalProperties: false, properties: { title: { type: "string" } } };
for await (const [text, meta] of meg.readPages("https://docs.megalania.io/start/quickstart", { schema })) console.log(meta.credits_charged, text.slice(0, 200));

Account, usage and stats

getAccount() shows your balance and limits, with either key. getDomainStats(host) shows how the whole platform does on one host. It is free. getUsage({ days: 7 }) and rotateWebhookSecret() need an account key (meg_ak_). The new secret shows once, so store it right away.

import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
console.log(await meg.getAccount());
console.log(await meg.getDomainStats("docs.megalania.io"));
const owner = new Megalania(process.env.MEG_ACCOUNT_KEY);
console.log(await owner.getUsage({ days: 7 }));
const secret = await owner.rotateWebhookSecret();

What you get back

scrape resolves to { headers, response }.

• response is the standard Response. Its body has not been read: call text(), arrayBuffer() for a screenshot, or read response.body as a stream for a large page. Each try gives up after timeout seconds (100 by default, new Megalania(key, { timeout })), body included, and throws a TimeoutError.
• headers holds the response headers under their own names, typed from the OpenAPI document: X-Request-Id, X-Credits-Charged (number), X-Engine, X-Target-Status (number), X-Timing-Target-Ms (number), X-Route, X-Geo, X-Identity, X-Billing (blocked on a free block, failure_cap when the actions failure cap charged it), X-Block, X-Challenge, X-Vendor, X-Hint (needs_js or browser_blocked), X-Untried, X-Hops (a JSON array of hops, like [{"rung":"h:dc","identity":"chrome","ms":412}]), X-Ladder-Truncated (true when your max_credits or balance removed routes), X-Truncated (max_credits or byte_cap when the page hit its byte limit), X-Network-Truncated (true when a cap cut the network part), X-Compact-Truncated (true when the cap of about 512 KiB cut compact_html), X-Usage-Warning (once per grant at 80% of its usage limit), Cache-Control, Content-Security-Policy (always the same sandbox policy), X-Content-Type-Options (always nosniff) and Referrer-Policy (always no-referrer). A header that is not present, or a number that does not parse, is missing from the object.

Errors

Any answer that is not a 2xx throws MegalaniaError. A page the site blocked is a free 422 with code blocked: you were not charged, and the error is the whole answer, with no page content. It has status, code (the error.code, such as bad_parameter), message, field (the parameter to change, on 400 and 422), requestId, reportUrl and headers (the typed headers a result has, those present, so a block shows X-Untried, X-Hops and X-Vendor). Error codes lists every code. If the body is not our JSON, for example a page from a proxy of yours, code is undefined and message holds the start of that body.

Retries

A 429 over_concurrency is retried once, after the seconds in Retry-After (1 when the header is missing), with the same Idempotency-Key. A 429 releases the key, so the retry is safe. A second 429 throws, and so does a 429 whose Retry-After is above 60 seconds or not a whole number of seconds, without waiting. Nothing else is retried: a 503 can mean the fetch ran, so you decide whether to send it again.