# Trawl

> Trawl is a hosted MCP server that gives AI agents a production-grade web-read tool with tiered anti-bot escalation. Connect once and your agent can scrape any URL, schedule recurring scrapes, trigger runs on demand, and retrieve structured JSON results from most publicly accessible pages — with no browser infrastructure to manage.

Trawl exposes 11 tools over Streamable HTTP (JSON-RPC 2.0) at https://api.trawl.me/api/mcp. Tool names are trawl_-prefixed — only the prefixed form is registered, there are no un-prefixed aliases. Tiered anti-bot escalation, auto-fix AI, and scheduled monitoring are handled server-side — your agent just calls tools. Free plan: 3,000 compute credits at signup, no credit card.

## MCP
- [MCP endpoint](https://api.trawl.me/api/mcp): Streamable HTTP, JSON-RPC 2.0, stateless POST. Auth: Authorization: Bearer <token> — a 7-day session JWT or a long-lived trawl_* API key, both available on every plan.
- [API & MCP reference](https://trawl.me/docs/api): full tool + endpoint docs incl. the MCP integration guide
- [CLI — @trawlme/cli](https://www.npmjs.com/package/@trawlme/cli): npm install -g @trawlme/cli && trawl login to get a free token

## Tools
- trawl_whoami: verify auth and return user, org, and scope
- trawl_health_ping: lightweight liveness probe (ok / version / uptime)
- trawl_list_scraps: list scraps accessible to this key (paginated, filterable)
- trawl_get_scrap: fetch a single scrap definition by id
- trawl_trigger_scrap_run: trigger an on-demand scrap run (honours rate-limit + cost guards)
- trawl_scrap_latest_result: get the most recent result for a scrap, of any status, without re-running it
- trawl_get_scrap_history: run history with timing, proxy tier, and fix version
- trawl_get_run: diagnose a single run by id: status, error, failed selector, proxy tier, and auto-fix outcome
- trawl_diagnose_scrap: diagnose a scrap by its newest run — same diagnosis as trawl_get_run, without listing history first; a scrap with no runs yet returns hasRuns: false and succeeds
- trawl_get_run_snapshot: a URL to the captured page HTML for a run, not the HTML itself (exceeds the tool payload budget); callable with an API key, though fetching the URL it returns needs a session token
- trawl_create_scrap: create a persistent, self-healing scrap from a URL + a goal — AI-generated code, first run + auto-fix included (metered ai:wizard, scope trawl:scraps:create)

## About
- [Trawl](https://trawl.me): hosted web scraping for AI agents, tiered anti-bot escalation included
- [Pricing](https://trawl.me/pricing): Free tier (3,000 compute, no card, scoped API keys included) + paid plans with renewing weekly quota, webhooks and the live JSON endpoint
- [Blog](https://blog.trawl.me): use cases and guides

## Get Started
- [Welcome](https://trawl.me/docs/get-started/welcome): **Turn any web page into structured JSON, on a schedule, with alerts when it changes.** That is Trawl. You describe what to watch and what to extract; Trawl runs a real headless browser (proxies, login, anti-bot handled for you), stores every run, and pushes the result to your dashboard, CLI, webhook, or AI agent.
- [Welcome (Markdown)](https://api.trawl.me/api/public/docs/welcome.md)
- [Quickstart](https://trawl.me/docs/get-started/quickstart): **Goal: your first structured JSON result in minutes.** You need two things — an API key and a scrap id — then one `curl` returns live data. The fastest way to get both is the setup below.
- [Quickstart (Markdown)](https://api.trawl.me/api/public/docs/quickstart.md)

## Build your scrap
- [Scraping](https://trawl.me/docs/build-your-scrap/scraping-basics): Trawl lets you build automated scrapers that extract data from any website, run on a schedule, and notify you when things change.
- [Scraping (Markdown)](https://api.trawl.me/api/public/docs/scraping-basics.md)
- [Scraping Advanced](https://trawl.me/docs/build-your-scrap/scraping-advanced): This guide covers advanced features: dynamic parameters, proxy tiers, and direct API access.
- [Scraping Advanced (Markdown)](https://api.trawl.me/api/public/docs/scraping-advanced.md)
- [Parameters](https://trawl.me/docs/build-your-scrap/parameters): This is the canonical reference for the `TRAWL.*` runtime namespace exposed to request scripts. It is linked directly from the Trawl web app (Settings → Parameters). For walkthroughs with code examples, see the **Scraping** and **Scraping Advanced** guides.
- [Parameters (Markdown)](https://api.trawl.me/api/public/docs/parameters.md)
- [Account Namespace](https://trawl.me/docs/build-your-scrap/account-sessions): This is the canonical reference for the `TRAWL.account.*` runtime namespace exposed to request scripts (the bare `account.*` global remains available as a legacy alias). It is linked directly from the Trawl web app (Settings → Account). For a hands-on walkthrough, see the **Scraping Advanced** guide.
- [Account Namespace (Markdown)](https://api.trawl.me/api/public/docs/account-sessions.md)
- [Security](https://trawl.me/docs/build-your-scrap/security): Trawl protects your credentials and integrations with layered controls — from the browser sandbox to your webhook receiver.
- [Security (Markdown)](https://api.trawl.me/api/public/docs/security.md)

## Run & monitor
- [Schedule](https://trawl.me/docs/run-monitor/scheduling): Automate your scraps so they run on their own — no manual clicks, no cron tabs on your server, just set it and forget it.
- [Schedule (Markdown)](https://api.trawl.me/api/public/docs/scheduling.md)
- [Notifications](https://trawl.me/docs/run-monitor/notifications): Stay in the loop without refreshing the dashboard. Trawl can email you results after every scheduled run and alert you when things break.
- [Notifications (Markdown)](https://api.trawl.me/api/public/docs/notifications.md)
- [AI Features](https://trawl.me/docs/run-monitor/ai-features): Trawl ships two AI-powered features that keep your scraps healthy and your data actionable — without manual intervention.
- [AI Features (Markdown)](https://api.trawl.me/api/public/docs/ai-features.md)
- [Data Quality](https://trawl.me/docs/run-monitor/data-quality): A scrap that runs is not the same as a scrap that works. This guide is about the second: how to tell a run that came back wrong from one that failed outright, why a scrap stops returning good data, and how to read the scrap list at a glance.
- [Data Quality (Markdown)](https://api.trawl.me/api/public/docs/data-quality.md)
- [Scrap Payload](https://trawl.me/docs/run-monitor/results-payload): The Scrap object is the same shape wherever you read it — REST response, MCP `trawl_get_scrap` tool, web app, webhook correlation. This guide documents the fields you can rely on.
- [Scrap Payload (Markdown)](https://api.trawl.me/api/public/docs/results-payload.md)

## Integrate
- [API Keys](https://trawl.me/docs/integrate/api-keys): Most users won't write raw HTTP — the **CLI** (`trawl login`, see [`@trawlme/cli`](https://www.npmjs.com/package/@trawlme/cli)) and **MCP server** (Claude Desktop / Cursor, see [MCP Server](/docs/use-with-claude/mcp)) cover day-to-day integration.
- [API Keys (Markdown)](https://api.trawl.me/api/public/docs/api-keys.md)
- [Webhooks](https://trawl.me/docs/integrate/webhooks): Receive real-time HTTP notifications when scraps change state — no polling required.
- [Webhooks (Markdown)](https://api.trawl.me/api/public/docs/webhooks.md)

## Use with Claude
- [CLI](https://trawl.me/docs/use-with-claude/cli): The `trawl` CLI is the fastest way to manage scraps from your terminal. Install once, authenticate once, then list, run and debug from anywhere — no browser needed. Pairs well with CI/CD pipelines and the Claude Code [skills](/docs/use-with-claude/skills).
- [CLI (Markdown)](https://api.trawl.me/api/public/docs/cli.md)
- [Skills](https://trawl.me/docs/use-with-claude/skills): Skills are pre-built Claude Code workflows that wrap common Trawl operations. Drop the `@trawlme/skills` package into your Claude Code session and Claude can scaffold scraps, manage sessions, capture banners, and run local tests — all driven by natural language. No boilerplate, no manual CLI incantations.
- [Skills (Markdown)](https://api.trawl.me/api/public/docs/skills.md)
- [MCP Server (Model Context Protocol)](https://trawl.me/docs/use-with-claude/mcp): Trawl exposes a Model Context Protocol (MCP) server so LLM clients — Claude Desktop, Cursor, n8n — can discover and call Trawl tools directly, without writing glue code.
- [MCP Server (Model Context Protocol) (Markdown)](https://api.trawl.me/api/public/docs/mcp.md)
- [Using Trawl with Claude](https://trawl.me/docs/use-with-claude/claude-usage): Trawl was built agent-first: every feature has a CLI command, a Claude Code skill, and an MCP tool. This guide walks you through the agent-first end-to-end story — from discovering a URL to scrape, to having Claude monitor it for you and iterate when things break.
- [Using Trawl with Claude (Markdown)](https://api.trawl.me/api/public/docs/claude-usage.md)

## Connect
Add Trawl to any MCP client (Claude Desktop, Cursor, or any JSON-RPC 2.0 client) — one Streamable-HTTP endpoint plus a Bearer token:

- url: https://api.trawl.me/api/mcp
- header: Authorization: Bearer YOUR_TOKEN

Free token: npm install -g @trawlme/cli && trawl login. Or create a long-lived API key under Developers → API Keys — available on every plan, Free included.