# TypeScript SDK

> Scrape, map, batches, monitors and tier from TypeScript or JavaScript, with typed headers and a typed error.

**TL;DR**: [install](#install) `megalania`, then `new Megalania().scrape({ url })`
(the key comes from `MEG_API_KEY` on Node). You get the `Response`, the headers as
typed fields, and a `MegalaniaError` with the error code and message. It
retries once on a 429, after `Retry-After`, and never on anything else. Every
scrape, map and create call carries an `Idempotency-Key`, so a repeat is never charged twice. No
runtime dependencies: it uses the built-in `fetch`.

The SDK is in beta. Its types are generated from the OpenAPI document
(`api/openapi/v1.yaml`), so they cannot drift from the API. The code is MIT
licensed at https://github.com/megalania/megalania-typescript. The
[Vercel AI SDK tools](/integrations/vercel-ai-sdk) build on it.

## Install

Node.js 22.18 or newer, Deno or Bun.

```sh
npm install megalania
```

## Scrape, extract and batch the Lab

Three first calls against the Lab, our test site: one page, rows from a list
page, and a batch of both. Each one runs as pasted once `MEG_API_KEY` is set.
The dashboard's **SDKs & code** page shows the same code.

**One page as Markdown**

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const page = await meg.scrape({ url: "https://lab.megalania.io/article", response_types: "markdown" });
console.log(page.headers["X-Credits-Charged"], (await page.response.text()).slice(0, 300));
```

**Rows from a list page**

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const rows = await meg.scrape({ url: "https://lab.megalania.io/cards", response_types: "json", extract: "items:.card{title:h2|price:.price|url:a@href}*" });
console.log(rows.headers["X-Credits-Charged"], (await rows.response.text()).slice(0, 300));
```

**Many pages in one batch**

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, { baseUrl: "https://get.megalania.io", apiUrl: "https://api.megalania.io" });
const batch = await meg.createBatch({ urls: ["https://lab.megalania.io/article", "https://lab.megalania.io/cards"], response_types: "markdown" });
console.log(batch.id, (await meg.getBatch(String(batch.id))).status);
```

## One page

Save as `scrape.mjs` and run `node scrape.mjs`:

```js
import { Megalania, MegalaniaError } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY, {
  baseUrl: process.env.MEG_BASE_URL ?? "https://get.megalania.io",
});
try {
  const { headers, response } = await meg.scrape({ url: "https://example.com/", country_code: "US" });
  console.log(headers["X-Credits-Charged"], headers["X-Target-Status"], headers["X-Request-Id"]);
  console.log((await response.text()).slice(0, 200));
} catch (e) {
  if (e instanceof MegalaniaError) console.log(e.status, e.code, e.field, e.message, e.requestId);
  throw e;
}
```

`scrape(envelope)` takes the [JSON envelope](/guides/fetch) as an object, typed as
`ScrapeEnvelope`. It sends only the keys you set. A key you leave out gets the
server's default, which is not always the same as sending that default
yourself: a `proxy_tier` you leave out lets us pick the cheapest route that
works.

`scrape` sends an `Idempotency-Key`: a fresh UUID, or the
`{ idempotencyKey: "..." }` you pass as the second argument. Pass your own on a
call you might send again yourself. A repeat is a 409 `replayed`, never a
second charge.

## Batches, monitors, map and tier

Batches and monitors live on the API host, `https://api.megalania.io`. Pass
`{ apiUrl }` to change it. Each method is named after its REST operation and
resolves to the JSON answer.

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
console.log(await meg.getTier("docs.megalania.io"));
console.log(await meg.mapSite({ url: "https://docs.megalania.io/start/quickstart", limit: 20 }));
const batch = await meg.createBatch({ urls: ["https://docs.megalania.io/start/quickstart"], response_types: "markdown" });
console.log(await meg.getBatch(batch.id));
console.log(await meg.listBatchItems(batch.id));
const monitor = await meg.createMonitor({ name: "Demo", urls: ["https://docs.megalania.io/start/quickstart"], every: "1d" });
await meg.updateMonitor(monitor.id, { paused: true });
```

| Method | REST call |
| -- | -- |
| `scrape(envelope)` | `POST /v1` |
| `getTier(host, query)` | `GET /v1/tier` |
| `getDomainStats(host, query)` | `GET /v1/stats/domain` |
| `getAccount()` | `GET /v1/account` |
| `getUsage(query)` | `GET /v1/account/usage` |
| `rotateWebhookSecret()` | `POST /v1/account/webhook-secret/rotate` |
| `mapSite(body)` | `POST /v1/map` |
| `createBatch(body)` | `POST /v1/batches` |
| `quoteBatch(body)` | `POST /v1/batches/quote` |
| `listBatches(query)` | `GET /v1/batches` |
| `getBatch(id)` | `GET /v1/batches/{id}` |
| `listBatchItems(id, query)` | `GET /v1/batches/{id}/items` |
| `cancelBatch(id)` | `POST /v1/batches/{id}/cancel` |
| `createMonitor(body)` | `POST /v1/monitors` |
| `listMonitors(query)` | `GET /v1/monitors` |
| `getMonitor(id)` | `GET /v1/monitors/{id}` |
| `updateMonitor(id, body)` | `PATCH /v1/monitors/{id}` |
| `deleteMonitor(id)` | `DELETE /v1/monitors/{id}` |
| `runMonitor(id)` | `POST /v1/monitors/{id}/run` |
| `listMonitorChecks(id, query)` | `GET /v1/monitors/{id}/checks` |

`scrape`, `mapSite`, `createBatch` and `createMonitor` send an
`Idempotency-Key`, yours or a fresh one. The 429 rule and `MegalaniaError` are
the same as for `scrape`.

## Read many pages

`waitBatch(id)` polls until a batch is done, failed or canceled.
`batchResults(id)` then yields each finished item with its body. It fetches
several result links at once, and it never sends your key to them.
`readPages(urls)` scrapes one page at a time, or crawls from one URL with
`{ crawl }`. It yields `[text, metadata]`. With a `schema`, it asks for
`json`, and the text is the filled schema.

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
const batch = await meg.createBatch({ urls: ["https://docs.megalania.io/start/quickstart"], response_types: "markdown" });
console.log((await meg.waitBatch(String(batch.id))).status);
for await (const [item, body] of meg.batchResults(String(batch.id))) console.log(item.url, body.length);
const schema = { type: "object", additionalProperties: false, properties: { title: { type: "string" } } };
for await (const [text, meta] of meg.readPages("https://docs.megalania.io/start/quickstart", { schema })) console.log(meta.credits_charged, text.slice(0, 200));
```

## Account, usage and stats

`getAccount()` shows your balance and limits, with either key.
`getDomainStats(host)` shows how the whole platform does on one host. It is
free. `getUsage({ days: 7 })` and `rotateWebhookSecret()` need an account key
(`meg_ak_`). The new secret shows once, so store it right away.

```js
import { Megalania } from "megalania";

const meg = new Megalania(process.env.MEG_API_KEY);
console.log(await meg.getAccount());
console.log(await meg.getDomainStats("docs.megalania.io"));
const owner = new Megalania(process.env.MEG_ACCOUNT_KEY);
console.log(await owner.getUsage({ days: 7 }));
const secret = await owner.rotateWebhookSecret();
```

## What you get back

`scrape` resolves to `{ headers, response }`.

- `response` is the standard `Response`. Its body has not been read: call
  `text()`, `arrayBuffer()` for a screenshot, or read `response.body` as a
  stream for a large page. Each try gives up after `timeout` seconds (100 by
  default, `new Megalania(key, { timeout })`), body included, and throws a
  `TimeoutError`.
- `headers` holds the response headers under their own names, typed from the
  OpenAPI document: `X-Request-Id`, `X-Credits-Charged` (number),
  `X-Engine`, `X-Target-Status` (number), `X-Timing-Target-Ms` (number),
  `X-Route`, `X-Geo`, `X-Identity`, `X-Billing` (`blocked` on a free block, `failure_cap` when the actions failure cap charged it),
  `X-Block`, `X-Challenge`, `X-Vendor`, `X-Hint` (`needs_js` or
  `browser_blocked`), `X-Untried`, `X-Hops`
  (a JSON array of hops, like `[{"rung":"h:dc","identity":"chrome","ms":412}]`), `X-Ladder-Truncated` (`true` when your
  `max_credits` or balance removed routes), `X-Truncated` (`max_credits`
  or `byte_cap` when the page hit its byte limit), `X-Network-Truncated` (`true` when a
  cap cut the `network` part), `X-Compact-Truncated` (`true` when the cap of
  about 512 KiB cut `compact_html`), `X-Usage-Warning` (once per grant
  at 80% of its usage limit), `Cache-Control`, `Content-Security-Policy` (always
  the same sandbox policy), `X-Content-Type-Options` (always `nosniff`) and
  `Referrer-Policy` (always `no-referrer`). A header that is not present, or a
  number that does not parse, is missing from the object.

## Errors

Any answer that is not a 2xx throws `MegalaniaError`. A page the site blocked
is a free 422 with `code` `blocked`: you were not charged, and the error is
the whole answer, with no page content. It has `status`,
`code` (the `error.code`, such as `bad_parameter`), `message`, `field` (the
parameter to change, on 400 and 422), `requestId`, `reportUrl` and `headers`
(the typed headers a result has, those present, so a block shows `X-Untried`, `X-Hops`
and `X-Vendor`).
[Error codes](/reference/errors) lists every code. If the body is not our JSON, for
example a page from a proxy of yours, `code` is `undefined` and `message`
holds the start of that body.

## Retries

A 429 `over_concurrency` is retried once, after the seconds in `Retry-After`
(1 when the header is missing), with the same `Idempotency-Key`. A 429 releases
the key, so the retry is safe. A second 429 throws, and so does a 429 whose
`Retry-After` is above 60 seconds or not a whole number of seconds, without
waiting. Nothing else is retried:
a 503 can mean the fetch ran, so you decide whether to send it again.