# Turning web pages into LLM-ready text

URL: https://grist.tools/guides/web-to-markdown-for-llms (updated 2026-09-28)

Turn a public web page into model context with `url-to-markdown` for article Markdown or `html-clean-text` for cleaned text and Markdown bounded by `max_chars`. Add `page-metadata`, `structured-data-extract` or `link-extract` when your agent also needs head metadata, embedded data or page links.

## Choose the content your model needs

Raw HTML includes navigation, scripts and markup that consume model tokens and require removal. Start by deciding whether the task needs the article, its declared metadata, embedded structured data or links to further pages. Make that choice before assembling the model context, and request the output that serves the task.

Use `url-to-markdown` when the article is the input you want. It fetches a page and returns article Markdown with headings, lists and resolved links, together with its title, byline and final URL. Its `max_chars` option cuts the returned Markdown by Unicode code points. Keep the final URL alongside the article when preparing the context.

Calls use `POST https://grist.tools/v1/<name>` with a JSON body and `content-type: application/json`. Read the free input schema at `GET /v1/<name>/schema` before building a request; `/openapi.json` also publishes the schemas.

## Use cleaned text when you need extraction controls

`html-clean-text` returns both `text` and `markdown`, plus declared metadata. Choose it when you need plain text, control over article extraction, inline HTML input or links from the extracted content. For fetching, supply `url` for an HTML, XHTML or plain-text page. For source you already hold, supply `html` instead; exactly one of those inputs is required.

Article extraction is enabled by default through `readability`. Disabling it converts the whole body, including navigation and footer. The same whole-body fallback applies when article extraction cannot find an article. Decide whether that broader content suits the task before passing the result to the model.

Set `max_chars` to bound both text and Markdown by Unicode code points; a longer result is cut and flagged as truncated. The example below uses `html-clean-text` on a sample article about measuring rainfall. It demonstrates fetching with article extraction, a character limit and `include_links` enabled.

## Select head metadata or embedded structured data

Choose `page-metadata` for fields declared in the page head: title, description, canonical URL, language, `hreflang`, favicon and Open Graph metadata. Its canonical, favicon, `hreflang` and `og:image` URLs are resolved to absolute against the final redirect destination or the page base; `og:url` and `twitter:image` are returned as authored, and a URL that cannot be resolved is `null`. Use these fields when the agent needs page identity and declared descriptions alongside, or instead of, article content.

Choose `structured-data-extract` for JSON-LD, Open Graph, microdata and meta tags, plus a `normalized` merge of the requested JSON-LD, Open Graph and meta sources; microdata is returned but not merged. Set `types` to select from `jsonld`, `opengraph`, `microdata` and `meta`, or omit it to request all of them. A source you do not request returns `null` and contributes nothing to the merge.

Broken JSON-LD scripts are skipped rather than causing extraction to throw. Inspect `jsonld_parse_errors` when reviewing a result, and keep the extracted sources available when using the merged fields. Ask for article content separately when the model also needs the prose.

## Choose links according to the next action

Enable `include_links` on `html-clean-text` when you need links inside the extracted content. Each comes with link text, `rel` and a same-host flag. Links resolve against the fetched URL or the supplied `base_url` for inline HTML; without a base for inline input, they remain as authored.

Choose `link-extract` for page links, with absolute URLs, `rel`, `nofollow` and internal-link flags. Set `internal_only` when you want links whose host equals the host of the final fetched page. The totals then describe that filtered set. Use those results to select further URLs for your agent to process.

The table below compares the endpoints and their declared prices and limits. Use the generated documentation links to inspect the contract for each call you plan to make.

## Services covered

| Endpoint | Price | Timeout | max_timeout_seconds | Size cap | Max duration |
| --- | --- | --- | --- | --- | --- |
| [url-to-markdown](https://grist.tools/docs/url-to-markdown) | $0.002 | 12 s | 33 | 1 MB (1,048,576 bytes) | - |
| [html-clean-text](https://grist.tools/docs/html-clean-text) | $0.002 | 12 s | 33 | 1 MB (1,048,576 bytes) | - |
| [page-metadata](https://grist.tools/docs/page-metadata) | $0.002 | 8 s | 29 | 3 MB (3,145,728 bytes) | - |
| [structured-data-extract](https://grist.tools/docs/structured-data-extract) | $0.002 | 8 s | 29 | 5 MB (5,242,880 bytes) | - |
| [link-extract](https://grist.tools/docs/link-extract) | $0.002 | 8 s | 29 | 5 MB (5,242,880 bytes) | - |

## Chain extraction around the question

For a task that needs an article and its declared identity, start with `page-metadata`, then send the page URL to `html-clean-text` or `url-to-markdown`. Keep the returned title and URL next to the chosen text representation. Add `structured-data-extract` only when the question also needs embedded data, such as the author or publication fields shown in its normalized output.

For a task that starts from an index page, call `link-extract`, select relevant returned URLs, and submit those URLs to the article endpoint. Inspect `links_truncated` before treating the returned link list as complete. If the task only needs references within the article, use `html-clean-text` with `include_links` and evaluate whether that already supplies the links you need.

Build the final context from the fields the task needs. Choose `text` or `markdown` from the cleaning response, inspect the truncation flag, and attach any selected metadata or structured fields. Keep the endpoint choice explicit at each stage so article extraction, metadata extraction and link selection remain deliberate decisions.

## Budget calls and allow for the payment deadline

A metadata and cleaned-text pipeline costs $0.002 plus $0.002 when both calls succeed. Adding structured data adds $0.002. An article-only path through `url-to-markdown` costs $0.002, while a discovery step through `link-extract` adds $0.002. Budget each selected page extraction separately when following links.

Payment uses x402 in USDC on Base, with no account or API key. A request without `payment-signature` receives HTTP 402 with the requirements, including the price (`amount`, in atomic units of the asset) and the whole-call `maxTimeoutSeconds`, before payment; the free `/schema` route reports the same value as `max_timeout_seconds`. Use those requirements when deciding whether to proceed with the call.

For `html-clean-text`, the handler timeout is 12 s, while the whole-call allowance in `max_timeout_seconds` is 33 seconds. That envelope includes payment verification and settlement around execution. Size the client deadline around the whole call.

Verification happens before execution; settlement follows handler success. Gate refusals and handler failures are never settled. If a settlement call fails without answering or its answer fails validation (that case is HTTP 500 `internal`, with a message starting "Payment settlement returned"), its outcome cannot be stated in advance, and the error message identifies that case. Preserve that message when deciding what to do next.

## Handle errors before choosing a retry

Use the typed code to decide what needs to change. `invalid_input` means the body does not match the schema, so correct the request. `blocked_target` means the network guard refused the destination. Check for a disallowed address, a scheme outside HTTP or HTTPS, a port outside the allowed web ports, or credentials in the URL. Repeating the unchanged request does not address those causes.

`unreachable_target` means the destination did not resolve, or the connection to it failed or closed before the body arrived. `upstream_status` means it answered with a status the endpoint cannot use; `upstream_timeout` means it did not answer within the declared timeout. Check the target and allow for a later retry when the underlying availability problem may have changed.

`unsupported_content_type` calls for a supported input type. `too_large` means the response exceeded the streaming size cap or the allowed decompression ratio, or the document exceeded the conversion limits of the endpoint on length, tag count or markup cost. `unprocessable` means the redirect chain could not be followed (too many hops or an unparseable `Location`), or bytes arrived but could not be processed, for example a page that yields no text or nests too deeply. Correct the source or endpoint choice where appropriate; reducing `max_chars` only changes returned text length.

`rate_limited` means too many calls from the same payer or toward the same destination domain, so reduce request pressure. Retryable HTTP 503 errors, including `concurrency_limit`, `registry_at_capacity` and `upstream_unavailable`, require waiting for the seconds specified by `retry-after`. When `upstream_unavailable` comes from the front proxy, the call may already have run and settled; check `authorizationState` before paying again. `internal` identifies a service fault; inspect the error message, including any settlement uncertainty, before retrying.

## Keep the extraction boundary visible

The fetched body cap for `html-clean-text` is 1 MB. Treat that input limit separately from `max_chars`, which bounds the returned text and Markdown. Check the generated limits for each endpoint you add to a chain; selecting another extraction purpose also means selecting its input contract.

`html-clean-text` refuses PDF and JSON content. Inline HTML uses `base_url` to resolve relative links, but `base_url` is not allowed together with `url`, where the final fetched URL supplies the base. Public URL fetching remains subject to the network guard, and `url-to-markdown` rechecks every redirect against private and reserved addresses. Keep unsupported content and refused destinations outside this extraction path.

## Questions

Q: Which endpoint should supply the text for my model context?
A: Use `url-to-markdown` for article Markdown with headings, lists, resolved links, title and byline. Choose `html-clean-text` when you want both plain text and Markdown, inline HTML input or control over article extraction. Select the returned representation that the task needs and keep its source URL alongside it.

Q: Can I clean HTML that my agent already has?
A: Yes, send it in the `html` field to `html-clean-text`, without `url`. Supply `base_url` if relative links need a base for resolution, and enable `include_links` if you need links from the extracted content. Without a base, those links remain as authored.

Q: Does `max_chars` set a model token budget?
A: `max_chars` counts Unicode code points, so use it as a character limit rather than a model token count. On `html-clean-text`, it bounds both text and Markdown, with longer output cut and flagged as truncated. On `url-to-markdown`, it cuts the returned Markdown.

Q: Why does cleaned output sometimes include navigation?
A: `html-clean-text` converts the whole body when `readability` is disabled or article extraction cannot find an article. That body includes navigation and footer content. Review whether the broader output suits your task before placing it in the model context.

Q: When should I request structured data instead of page metadata?
A: Choose `structured-data-extract` when you need JSON-LD, microdata or the normalized merge of embedded sources. Use `types` to select the sources you need; unrequested sources return `null` and are excluded from the merge. Choose `page-metadata` for head fields such as canonical URL, language, `hreflang` and favicon.

Q: Should an agent retry every failed extraction?
A: No; inspect the typed error and correct invalid input, refused targets or unsuitable content before repeating the request. For retryable HTTP 503 errors, wait for the interval in `retry-after`. An upstream availability failure may warrant a later retry, while a settlement call that failed without answering or an answer that failed validation (that case is HTTP 500 `internal`, with a message starting "Payment settlement returned") requires attention to the uncertainty stated in the error message.

Q: Do I need an account before checking the price and schema?
A: No account or API key is required, and `GET /v1/<name>/schema` is free. A call without `payment-signature` returns HTTP 402 with the price in `amount` and the whole-call deadline in `maxTimeoutSeconds` before you pay. Successful calls settle through x402 in USDC on Base after the handler completes.

This page as HTML: https://grist.tools/guides/web-to-markdown-for-llms. Every page of grist.tools has a Markdown version at its address plus .md.
