---
name: scrape
description: "Turn any public web page into clean, LLM-ready Markdown, raw HTML or JSON fields picked with CSS selectors. Optional JavaScript rendering, screenshots and an automatic residential-proxy fallback when a site blocks. Use when the user wants what a web page says: read, summarize or quote an article or docs page as clean Markdown, get the raw HTML, or pick fields such as prices, headings or links with CSS selectors. Up to 20 URLs per run. Calls the Pullwire API with the key in PULLWIRE_API_KEY; costs from 1 credit per scraped page."
metadata:
  openclaw:
    requires:
      env:
        - PULLWIRE_API_KEY
      bins:
        - curl
    primaryEnv: PULLWIRE_API_KEY
    homepage: https://pullwire.dev/tools/scrape
---

# Web Scraper

Turn any public web page into clean, LLM-ready Markdown, raw HTML or JSON fields picked with CSS selectors. Optional JavaScript rendering, screenshots and an automatic residential-proxy fallback when a site blocks. The data comes from the Pullwire API: you send the input as JSON and get structured JSON back. Only public data.

## Setup (once)

The API key must be in the environment variable `PULLWIRE_API_KEY`. Keys start with `af_live_`. Check it without printing it:

```bash
test -n "$PULLWIRE_API_KEY" && echo "key is set" || echo "PULLWIRE_API_KEY is missing"
```

If it is missing, stop and tell the user: sign up at https://pullwire.dev/login (new accounts get 2,000 free credits), create a key at https://pullwire.dev/app/keys, then run `export PULLWIRE_API_KEY="af_live_..."` in the shell the agent uses. Never print the key, write it into a file or repeat it in the chat.

## Run it

```bash
curl -sS -X POST "https://pullwire.dev/api/v1/tools/scrape/runs" \
  -H "x-api-key: $PULLWIRE_API_KEY" \
  -H "content-type: application/json" \
  -d '{"urls":["https://example.com/"]}'
```

The request waits up to 25 seconds and answers with `{"run": {...}, "results": [...]}`. Check `run.status` (`succeeded` or `failed`) and `run.error`: when it is set, the run stopped early and `results` is partial (everything in it was charged). `run.creditsUsed` is what the run cost. For big answers add `-o result.json` and read only the fields you need.

Good to know:

- Pass several pages in `urls` (up to 20) instead of one run per page; every page is one result.
- `format: "json"` needs `extract`: field name -> CSS selector, or `{"selector": ..., "attribute": "href", "all": true}` for an attribute or every match.
- A page behind a bot check (Cloudflare, DataDome and similar) fails with `BLOCKED` and costs nothing. With `proxy: "auto"` (default) it is retried through a residential proxy where the service has one.
- `renderJs` and `screenshot` run in a headless browser on our render workers. A call waits up to about 50 seconds for the page; if it takes longer, the API answers `202` with the run id (poll `GET /runs/{id}`). Where no browser is connected the run answers `UNAVAILABLE` and costs nothing. Without it, pages that build their text with JavaScript come back almost empty.
- `truncated: true` means the page was cut (very large page or more than 1,000,000 characters of content); say so when you quote it.

## Input

The JSON body is the tool input. Fields:

| Field | Type | Required | Default | What it does |
| --- | --- | --- | --- | --- |
| `url` | string | no |  | URL: The page to scrape. |
| `urls` | string[] | no |  | URLs: Several pages in one run (up to 20); one result per page. |
| `format` | "markdown" \| "html" \| "json" | no | `"markdown"` | Output format: markdown: clean, LLM-ready text of the page. html: the page HTML. json: the fields from `extract`. |
| `onlyMainContent` | boolean | no | `true` | Main content only: Markdown: drop navigation, cookie banners, sidebars and footers. |
| `extract` | object | no |  | Extraction rules: For format json: field name -> CSS selector, or {selector, attribute, all}. attribute reads e.g. href/src/content ("html" = inner HTML); all returns every match. |
| `renderJs` | boolean | no | `false` | Render JavaScript: Load the page in a headless browser first (for sites that build their content with JS). |
| `waitFor` | string | no |  | Wait for: With renderJs: a CSS selector to wait for, or a delay in milliseconds (e.g. 2000, max 15000). |
| `proxy` | "auto" \| "none" \| "residential" | no | `"auto"` | Proxy: auto: direct first, residential proxy only if the site blocks us. none: never a proxy. residential: always. |
| `country` | string | no |  | Country: Exit country for the residential proxy (two-letter code). |
| `timeoutMs` | integer | no | `20000` | Timeout (ms): Per page, for loading (and rendering). |
| `screenshot` | boolean | no | `false` | Screenshot: With renderJs: a PNG of the viewport, returned as a signed download URL (valid 7 days). |

A run returns at most one result per input entry. Up to 25 credits per result (the most one can cost) are reserved before the run starts and the unused part is given back.

## Output

Each item in `results` has these fields:

| Field | Type | Always there |
| --- | --- | --- |
| `url` | string | yes |
| `finalUrl` | string | yes |
| `statusCode` | integer | yes |
| `format` | "markdown" \| "html" \| "json" | yes |
| `title` | string | no |
| `description` | string | no |
| `language` | string | no |
| `content` | string | no |
| `data` | object | no |
| `contentType` | string | no |
| `truncated` | boolean | yes |
| `renderedJs` | boolean | yes |
| `proxyClass` | "none" \| "datacenter" \| "residential" | yes |
| `country` | string | no |
| `screenshotUrl` | string | no |
| `screenshotExpiresAt` | string | no |
| `bytes` | integer | yes |
| `browserMs` | integer | no |
| `durationMs` | integer | yes |
| `credits` | integer | yes |
| `scrapedAt` | string | yes |

## Cost

From 1 credit per scraped page. One successfully scraped page: 1 credit, +2 with JavaScript rendering, +2 when the residential proxy was needed, +1 for a screenshot (very large pages through a proxy cost more). Failed pages are free. Check the balance with:

```bash
curl -sS "https://pullwire.dev/api/v1/me" -H "x-api-key: $PULLWIRE_API_KEY"
```

The answer has `balance` in credits. Tell the user what a run cost when it was more than a few credits.

## Bigger jobs

Add `?async=1` to the run URL to start a background run (up to 50 seconds). It answers `202` with `run.id` right away. Poll every few seconds until `status` is `succeeded` or `failed`, then fetch the results:

```bash
curl -sS "https://pullwire.dev/api/v1/runs/$RUN_ID" -H "x-api-key: $PULLWIRE_API_KEY"
curl -sS "https://pullwire.dev/api/v1/runs/$RUN_ID/results" -H "x-api-key: $PULLWIRE_API_KEY"
```

Add `?format=csv` to the results URL when the user wants a spreadsheet.

## Errors

Errors answer `{"error": {"code": "...", "message": "..."}}`.

| Status | Code | What to do |
| --- | --- | --- |
| 400 | `INVALID_INPUT` | The message names the field. Fix the input and retry; nothing was charged. |
| 401 | `UNAUTHORIZED` | The key is missing, wrong or revoked. Ask the user to set a valid key in `PULLWIRE_API_KEY`. |
| 402 | `INSUFFICIENT_CREDITS` | The balance cannot cover the reservation. Lower `maxResults` or tell the user the balance is too low. |
| 404 | `NOT_FOUND` | Unknown tool or run ID. |
| 429 | `RATE_LIMITED` | Used up its burst of 120 requests and its 60 per minute. Wait, then retry. |
| 500 | `INTERNAL_ERROR` | Retry later. If the body has `runId`, fetch the partial results with `GET /runs/{runId}/results`. |
| 503 | `TOOL_UNAVAILABLE` | Our latest automatic checks against this site failed, so its runs are paused. Nothing was reserved or charged. Tell the user and try again later. |
| 503 | `SERVICE_UNAVAILABLE` | Retry later. |

Docs: https://pullwire.dev/docs · Tool page: https://pullwire.dev/tools/scrape