# Sitemap Parse API

> Parses a sitemap, a sitemap index or a bare origin into a flat, bounded list of URLs with lastmod, changefreq and priority.

- Endpoint: `POST https://grist.tools/v1/sitemap-parse`
- Price: $0.002 per successful call, paid via x402 in USDC on Base
- Timeout: 14 s; max_timeout_seconds in the 402: 35 (the timeout plus payment verification and settlement)
- Response cap: 10 MB (10,485,760 bytes)
- Family: [Feeds & site structure](https://grist.tools/tools/feeds)
- Input schema, free: https://grist.tools/v1/sitemap-parse/schema
- Updated: 2026-11-13

## When to use it

Needs the network, follows a sitemap index across many documents and normalises four different legal sitemap formats. An agent that does it itself either writes an XML parser with the XXE and fan-out defences, or fetches an unbounded tree of documents a stranger chose.

## When not to use it

- The input is over 10 MB (10,485,760 bytes): sitemap-parse answers too_large (413).
- The destination needs more than 14 s to answer: the call ends with upstream_timeout (424).
- The destination is on a private or local network: the network guard refuses it with blocked_target (403).

## Parameters

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | yes | Absolute http(s) URL, up to 2048 characters, of a sitemap, a sitemap index or a bare origin (path / and no query); a bare origin is resolved through robots.txt Sitemap: lines or /sitemap.xml. |
| `limit` | integer | no | Maximum number of URLs returned across every document read, 1 to 50000 (default 5000); past it the list is cut and reported as truncated. Default: 5000. |
| `include_metadata` | boolean | no | When true each URL carries its lastmod, changefreq and priority; when false (default) those three fields are null. Default: false. |
| `max_sitemaps` | integer | no | Maximum number of sitemap documents fetched, entry document included, 1 to 50 (default 10); children past it are reported as skipped with reason sitemap_limit. Default: 10. |
| `discover_from_robots` | boolean | no | Only for a bare origin: when true (default) the Sitemap: lines of /robots.txt are read first with /sitemap.xml as the fallback, when false /sitemap.xml is used directly. Default: true. |

## Limits

| Limit | Value |
| --- | --- |
| Size cap | 10 MB (10,485,760 bytes) |
| Timeout | 14 s |

## Example

Request:

```http
POST /v1/sitemap-parse
content-type: application/json
payment-signature: <base64 x402 payload>

{
  "url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
  "limit": 100,
  "include_metadata": true
}
```

Response (200 OK, with a payment-response settlement receipt header):

```json
{
  "requested_url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
  "entry_url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
  "source_url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
  "discovery": "direct",
  "kind": "mixed",
  "format": "xml",
  "urls": [
    {
      "loc": "https://grist.tools/samples/web/fieldnotes/index.html",
      "lastmod": "2026-08-25T00:00:00Z",
      "changefreq": "daily",
      "priority": 1
    },
    {
      "loc": "https://grist.tools/samples/web/fieldnotes/about.html",
      "lastmod": "2026-08-20T00:00:00Z",
      "changefreq": "yearly",
      "priority": 0.3
    },
    {
      "loc": "https://grist.tools/samples/web/fieldnotes/notes/first-frost.html",
      "lastmod": "2026-08-24T06:30:00Z",
      "changefreq": "monthly",
      "priority": 0.6
    },
    {
      "loc": "https://grist.tools/samples/web/fieldnotes/notes/rain-gauge.html",
      "lastmod": "2026-08-25T06:15:00Z",
      "changefreq": null,
      "priority": 0.6
    }
  ],
  "count": 4,
  "total_seen": 4,
  "truncated": false,
  "stopped_reason": null,
  "sitemap_count": 3,
  "sitemaps_visited": [
    {
      "url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
      "kind": "sitemapindex",
      "url_count": 0,
      "http_status": 200,
      "bytes": 407,
      "gzip": false,
      "depth": 0
    },
    {
      "url": "https://grist.tools/samples/web/fieldnotes/sitemap-pages.xml",
      "kind": "urlset",
      "url_count": 2,
      "http_status": 200,
      "bytes": 479,
      "gzip": false,
      "depth": 1
    },
    {
      "url": "https://grist.tools/samples/web/fieldnotes/sitemap-notes.xml",
      "kind": "urlset",
      "url_count": 2,
      "http_status": 200,
      "bytes": 493,
      "gzip": false,
      "depth": 1
    }
  ],
  "sitemaps_skipped": [],
  "sitemaps_skipped_total": 0,
  "sitemaps_skipped_by_reason": {
    "depth_limit": 0,
    "sitemap_limit": 0,
    "cross_host": 0,
    "too_large": 0,
    "malformed": 0,
    "duplicate": 0,
    "time_budget": 0,
    "not_fetchable": 0
  },
  "invalid_loc_count": 0,
  "invalid_lastmod_count": 0,
  "robots_sitemaps": null,
  "fetched_at": "2026-09-01T12:00:00.000Z"
}
```

Sample file: web/fieldnotes/sitemap-index.xml. Sitemap index of the Field Notes sample site pointing at its two urlset sitemaps. Sample file: web/fieldnotes/sitemap-pages.xml. Urlset sitemap listing the home and about pages of the Field Notes sample site, with lastmod, changefreq and priority. Sample file: web/fieldnotes/sitemap-notes.xml. Urlset sitemap listing the two Field Notes article pages, with lastmod in two time zones and priority. Sample file: web/fieldnotes/index.html. Sample file: web/fieldnotes/about.html. About page of the Field Notes sample site, a short plain HTML page. Sample file: web/fieldnotes/notes/first-frost.html. Sample file: web/fieldnotes/notes/rain-gauge.html. Plain Field Notes article page with only a title, a description meta tag and an icon link.

Response fields:

| Field | Type | In the example |
| --- | --- | --- |
| `requested_url` | string | "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml" |
| `entry_url` | string | "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml" |
| `source_url` | string | "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml" |
| `discovery` | string | "direct" |
| `kind` | string | "mixed" |
| `format` | string | "xml" |
| `urls` | array | 4 items, each with loc, lastmod, changefreq and priority |
| `count` | number | 4 |
| `total_seen` | number | 4 |
| `truncated` | boolean | false |
| `stopped_reason` | null | null |
| `sitemap_count` | number | 3 |
| `sitemaps_visited` | array | 3 items, each with url, kind, url_count, http_status, bytes, gzip and depth |
| `sitemaps_skipped` | array | 0 items |
| `sitemaps_skipped_total` | number | 0 |
| `sitemaps_skipped_by_reason` | object | an object with depth_limit, sitemap_limit, cross_host, too_large, malformed, duplicate, time_budget and not_fetchable |
| `invalid_loc_count` | number | 0 |
| `invalid_lastmod_count` | number | 0 |
| `robots_sitemaps` | null | null |
| `fetched_at` | string | "2026-09-01T12:00:00.000Z" |

## Call it from code

```js
// sitemap-parse: $0.002 per call, paid in USD Coin on eip155:8453 via x402
// Install: npm install @x402/fetch@2 @x402/evm@2 viem@2
// Save as client.mjs (ES module, Node 18+), export EVM_PRIVATE_KEY with the paying wallet's key in your shell, then run: node client.mjs
// The wrapper reads the 402 (payment-required), signs and retries with payment-signature.
import { wrapFetchWithPayment, x402Client, decodePaymentResponseHeader } from '@x402/fetch'
import { ExactEvmScheme } from '@x402/evm/exact/client'
import { privateKeyToAccount } from 'viem/accounts'

const account = privateKeyToAccount(process.env.EVM_PRIVATE_KEY)
const client = new x402Client().register('eip155:8453', new ExactEvmScheme(account))
const fetchWithPayment = wrapFetchWithPayment(fetch, client)

const body = {
  "url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
  "limit": 100,
  "include_metadata": true
}

const res = await fetchWithPayment('https://grist.tools/v1/sitemap-parse', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify(body),
})
console.log(res.status, await res.json())
const settle = res.headers.get('payment-response')
if (settle) console.log(decodePaymentResponseHeader(settle).transaction)
```

```python
# sitemap-parse: $0.002 per call, paid in USD Coin on eip155:8453 via x402
# Install: pip install "x402[requests,evm]>=2.24,<3"
# The session reads the 402 (payment-required), signs and retries with payment-signature.
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http import decode_payment_response_header
from x402.http.clients.requests import x402_requests
from x402.mechanisms.evm.exact import register_exact_evm_client
from x402.mechanisms.evm.signers import EthAccountSigner

account = Account.from_key(os.environ["EVM_PRIVATE_KEY"])
client = x402ClientSync()
register_exact_evm_client(client, EthAccountSigner(account), networks="eip155:8453")
session = x402_requests(client)

payload = {
    "url": "https://grist.tools/samples/web/fieldnotes/sitemap-index.xml",
    "limit": 100,
    "include_metadata": True
}

res = session.post("https://grist.tools/v1/sitemap-parse", json=payload)
print(res.status_code, res.json())
settle = res.headers.get("payment-response")
if settle:
    print(decode_payment_response_header(settle).transaction)
```

```sh
# sitemap-parse: $0.002 per call, paid in USD Coin on eip155:8453 via x402
# 1. Unpaid call: HTTP 402, the requirements in the payment-required header (base64 JSON) and in the body.
curl -i -X POST 'https://grist.tools/v1/sitemap-parse' -H 'content-type: application/json' -d '{"url":"https://grist.tools/samples/web/fieldnotes/sitemap-index.xml","limit":100,"include_metadata":true}'

# 2. Same call with the signed payment (an EIP-3009 authorization, EIP-712 signed: it cannot be
#    typed by hand). x-payment is accepted as the v1 alternative. HTTP 200 carries payment-response.
curl -i -X POST 'https://grist.tools/v1/sitemap-parse' -H 'content-type: application/json' -H 'payment-signature: <base64 x402 payload>' -d '{"url":"https://grist.tools/samples/web/fieldnotes/sitemap-index.xml","limit":100,"include_metadata":true}'
```

## Errors

Every error is typed JSON. Only a successful call is settled.

| Code | HTTP | When |
| --- | --- | --- |
| `invalid_input` | 400 | The body does not match the schema on this page. Nothing is charged. |
| `blocked_target` | 403 | The destination is refused by the network guard: a disallowed address, or, for HTTP requests, a scheme other than http/https, a port other than 80/443, or credentials in the URL. |
| `unreachable_target` | 424 | The destination you named did not resolve or refused the connection. |
| `upstream_status` | 424 | The destination you named answered with a status this endpoint cannot use. |
| `upstream_timeout` | 424 | The destination you named did not answer before the declared timeout for this endpoint expired. |
| `unsupported_content_type` | 415 | The destination returned a type this endpoint does not handle. |
| `too_large` | 413 | The response exceeded the size cap while streaming, or decompressed at more than the allowed ratio. |
| `unprocessable` | 422 | The bytes arrived but could not be processed, for instance a malformed document. |
| `internal` | 500 | Our fault, and in almost every case nothing is charged. The exceptions come after the call has run and the settlement call was made: the message "The work was done but settlement could not be completed", and the messages starting with "Payment settlement returned". There, whether the payment moved is unknown to us as well. The payment moves at most once: its nonce is single-use and settlement is idempotent, so a retry with it is never charged twice. A retry is normally refused with a 402: replay_consumed, an expiry reason, or a reason saying the nonce was already used once the transfer is on-chain. Only if our server restarted before the transfer was final can a retry run the call again. Even then, the payment moves at most once. To see whether it was settled, read the EIP-3009 authorizationState of its nonce on-chain; to call again, sign a new payment. |
| `rate_limited` | 429 | Too many calls from the same payer, or too many calls towards the same destination domain. Nothing is charged. |
| `concurrency_limit`, `registry_at_capacity`, `upstream_unavailable` | 503 | We could not serve the call right now. Retryable: wait the number of seconds in the retry-after header, then repeat the call. concurrency_limit and registry_at_capacity mean we were at capacity: the call did not run. You can send the same payment again if its validBefore is still far enough away to cover another call. If it is not, the retry gets a 402 authorization_validity_too_short: sign a new payment. upstream_unavailable means our internal service failed or did not answer in time: the call may already have run and the payment may have been settled. A retry with the same payment can get a 402 whose error field says why. authorization_validity_too_short: too little time is left before validBefore to cover another call. This check comes before the replay check, so it does not tell whether the first payment was settled. replay_consumed (the payment was used and may have been settled) or replay_exhausted (too many attempts): sign a new payment. replay_in_flight: the first call is still running, so wait and do not pay again. invalid_exact_evm_nonce_already_used: the transfer is already on-chain, so this payment cannot move again. Sign a new payment. An authorization can be settled only once, since its nonce is single-use. To see whether it was settled, read the EIP-3009 authorizationState of its nonce on-chain. |
| `misconfigured` | 503 | The service is not configured to serve calls. The call did not run and nothing is charged. It needs a fix on our side, so there is no retry delay to wait for. |

## Questions

### How much does a call to the Sitemap Parse API cost?

$0.002 per call, paid in USD Coin on Base via x402. There is no account and no API key: the price is in the 402 answer, before anything is paid.

### What is the timeout of sitemap-parse?

The endpoint itself runs for at most 14 s. max_timeout_seconds in the 402 answer is 35: that covers the whole call, including payment verification and settlement around it.

### How large can a response from sitemap-parse be?

The response-size cap is 10 MB (10,485,760 bytes).

### Is the sitemap-parse input schema free to read?

Yes. GET https://grist.tools/v1/sitemap-parse/schema returns the input schema for free, paid endpoints included.

### What happens if a call to sitemap-parse fails?

The payment is verified before the handler runs and settled only after it succeeds, so a call refused by the gate, or one that fails inside the handler, is never settled. The single case that cannot be stated in advance is a settlement call that fails without answering: the error message says so when that is what happened.

### What does sitemap-parse return?

A JSON object with 20 top-level fields: requested_url, entry_url, source_url, discovery, kind, format, urls, count, total_seen, truncated, stopped_reason, sitemap_count, sitemaps_visited, sitemaps_skipped, sitemaps_skipped_total, sitemaps_skipped_by_reason, invalid_loc_count, invalid_lastmod_count, robots_sitemaps and fetched_at. The example on this page is a real call over web/fieldnotes/sitemap-index.xml, web/fieldnotes/sitemap-pages.xml, web/fieldnotes/sitemap-notes.xml, web/fieldnotes/index.html, web/fieldnotes/about.html, web/fieldnotes/notes/first-frost.html and web/fieldnotes/notes/rain-gauge.html.

## Related

- [Feeds & site structure APIs](https://grist.tools/tools/feeds)
- [How AI agents pay for APIs with x402](https://grist.tools/guides/x402-payments-for-ai-agents)
- [All APIs](https://grist.tools/docs)

This page as HTML: https://grist.tools/docs/sitemap-parse. Every page of grist.tools has a Markdown version at its address plus .md.
