# Trawl

> Trawl is a hosted MCP server that gives AI agents a production-grade, stealth-tier web-read tool. Connect once and your agent can scrape any URL, schedule recurring scrapes, trigger runs on demand, and retrieve structured JSON results — including from sites protected by DataDome and similar anti-bot systems — with no browser infrastructure to manage.

Trawl exposes 9 tools over Streamable HTTP (JSON-RPC 2.0) at https://api.trawl.me/api/mcp. Tool names are trawl_-prefixed (the un-prefixed forms are kept as aliases). Anti-bot stealth, auto-fix AI, and scheduled monitoring are handled server-side — your agent just calls tools. Free plan: 500 compute credits at signup, no credit card.

## MCP
- [MCP endpoint](https://api.trawl.me/api/mcp): Streamable HTTP, JSON-RPC 2.0, stateless POST. Auth: Authorization: Bearer <token> (7-day session JWT free / long-lived trawl_* API key Pro).
- [API & MCP reference](https://trawl.me/docs/api): full tool + endpoint docs incl. the MCP integration guide
- [CLI — @trawlme/cli](https://www.npmjs.com/package/@trawlme/cli): npm install -g @trawlme/cli && trawl login to get a free token

## Tools
- trawl_whoami: verify auth and return user, org, and scope
- trawl_health_ping: lightweight liveness probe (ok / version / uptime)
- trawl_list_scraps: list scraps accessible to this key (paginated, filterable)
- trawl_get_scrap: fetch a single scrap definition by id
- trawl_trigger_scrap_run: trigger an on-demand scrap run (honours rate-limit + cost guards)
- trawl_scrap_latest_result: get the most recent successful result for a scrap
- trawl_get_scrap_history: run history with timing, cost, proxy tier, and fix version
- trawl_list_tasks: list scheduled monitoring tasks (org-scoped)
- trawl_get_task: fetch a single task by id

## About
- [Trawl](https://trawl.me): hosted stealth-tier web scraping for AI agents
- [Pricing](https://trawl.me/pricing): Free tier (500 compute, no card) + paid plans with long-lived API keys + extended quota
- [Blog](https://blog.trawl.me): use cases and guides

## Get Started
- [Welcome](https://trawl.me/docs/get-started/welcome): **Turn any web page into structured JSON, on a schedule, with alerts when it changes.** That is Trawl. You describe what to watch and what to extract; Trawl runs a real headless browser (proxies, login, anti-bot handled for you), stores every run, and pushes the result to your dashboard, CLI, webhook, or AI agent.
- [Welcome (Markdown)](https://trawl.me/docs/get-started/welcome.md)
- [Quickstart](https://trawl.me/docs/get-started/quickstart): **Goal: your first structured JSON result in under a minute.** You need two things — an API key and a scrap id — then one `curl` returns live data. The fastest way to get both is the 60-second setup below.
- [Quickstart (Markdown)](https://trawl.me/docs/get-started/quickstart.md)

## Build your scrap
- [Scraping](https://trawl.me/docs/build-your-scrap/scraping-basics): Trawl lets you build automated scrapers that extract data from any website, run on a schedule, and notify you when things change.
- [Scraping (Markdown)](https://trawl.me/docs/build-your-scrap/scraping-basics.md)
- [Scraping Advanced](https://trawl.me/docs/build-your-scrap/scraping-advanced): This guide covers advanced features: dynamic parameters, proxy tiers, data validation, and direct API access.
- [Scraping Advanced (Markdown)](https://trawl.me/docs/build-your-scrap/scraping-advanced.md)
- [Parameters](https://trawl.me/docs/build-your-scrap/parameters): This is the canonical reference for the `TRAWL.*` runtime namespace exposed to request scripts. It is linked directly from the Trawl web app (Settings → Parameters). For walkthroughs with code examples, see the **Scraping** and **Scraping Advanced** guides.
- [Parameters (Markdown)](https://trawl.me/docs/build-your-scrap/parameters.md)
- [Account Namespace](https://trawl.me/docs/build-your-scrap/account-sessions): This is the canonical reference for the `TRAWL.account.*` runtime namespace exposed to request scripts (the bare `account.*` global remains available as a legacy alias). It is linked directly from the Trawl web app (Settings → Account). For a hands-on walkthrough, see the **Scraping Advanced** guide.
- [Account Namespace (Markdown)](https://trawl.me/docs/build-your-scrap/account-sessions.md)
- [Security](https://trawl.me/docs/build-your-scrap/security): Trawl protects your credentials and integrations with layered controls — from the browser sandbox to your webhook receiver.
- [Security (Markdown)](https://trawl.me/docs/build-your-scrap/security.md)

## Run & monitor
- [Schedule](https://trawl.me/docs/run-monitor/scheduling): Automate your scraps so they run on their own — no manual clicks, no cron tabs on your server, just set it and forget it.
- [Schedule (Markdown)](https://trawl.me/docs/run-monitor/scheduling.md)
- [Notifications](https://trawl.me/docs/run-monitor/notifications): Stay in the loop without refreshing the dashboard. Trawl can email you results after every scheduled run and alert you when things break.
- [Notifications (Markdown)](https://trawl.me/docs/run-monitor/notifications.md)
- [AI Features](https://trawl.me/docs/run-monitor/ai-features): Trawl ships two AI-powered features that keep your scraps healthy and your data actionable — without manual intervention.
- [AI Features (Markdown)](https://trawl.me/docs/run-monitor/ai-features.md)
- [Data Quality](https://trawl.me/docs/run-monitor/data-quality): This is the canonical reference for data validation on scrap results. It is linked directly from the Trawl web app (Settings → Data Validation). For a hands-on walkthrough, see the **Scraping Advanced** guide.
- [Data Quality (Markdown)](https://trawl.me/docs/run-monitor/data-quality.md)
- [Scrap Payload](https://trawl.me/docs/run-monitor/results-payload): The Scrap object is the same shape wherever you read it — REST response, MCP `get_scrap` tool, web app, webhook correlation. This guide documents the fields you can rely on.
- [Scrap Payload (Markdown)](https://trawl.me/docs/run-monitor/results-payload.md)

## Integrate
- [API Keys](https://trawl.me/docs/integrate/api-keys): Most users won't write raw HTTP — the **CLI** (`trawl login`, see [`trawl_cli`](https://github.com/comes-io/trawl_cli)) and **MCP server** (Claude Desktop / Cursor, see [MCP Server](/docs/use-with-claude/mcp)) cover day-to-day integration. API keys are the shared authentication layer underneath both, and the direct path when you need plain REST.
- [API Keys (Markdown)](https://trawl.me/docs/integrate/api-keys.md)
- [Webhooks](https://trawl.me/docs/integrate/webhooks): Receive real-time HTTP notifications when scraps change state — no polling required.
- [Webhooks (Markdown)](https://trawl.me/docs/integrate/webhooks.md)

## Use with Claude
- [CLI](https://trawl.me/docs/use-with-claude/cli): The `trawl` CLI is the fastest way to manage scraps from your terminal. Install once, authenticate once, then list, run, watch, and debug from anywhere — no browser needed. Pairs well with CI/CD pipelines and the Claude Code [skills](/docs/use-with-claude/skills).
- [CLI (Markdown)](https://trawl.me/docs/use-with-claude/cli.md)
- [Skills](https://trawl.me/docs/use-with-claude/skills): Skills are pre-built Claude Code workflows that wrap common Trawl operations. Drop the `@trawlme/skills` package into your Claude Code session and Claude can scaffold scraps, manage sessions, capture banners, and run local tests — all driven by natural language. No boilerplate, no manual CLI incantations.
- [Skills (Markdown)](https://trawl.me/docs/use-with-claude/skills.md)
- [MCP Server (Model Context Protocol)](https://trawl.me/docs/use-with-claude/mcp): Trawl exposes a Model Context Protocol (MCP) server so LLM clients — Claude Desktop, Cursor, n8n — can discover and call Trawl tools directly, without writing glue code.
- [MCP Server (Model Context Protocol) (Markdown)](https://trawl.me/docs/use-with-claude/mcp.md)
- [Using Trawl with Claude](https://trawl.me/docs/use-with-claude/claude-usage): Trawl was built agent-first: every feature has a CLI command, a Claude Code skill, and an MCP tool. This guide walks you through the agent-first end-to-end story — from discovering a URL to scrape, to having Claude monitor it for you and iterate when things break.
- [Using Trawl with Claude (Markdown)](https://trawl.me/docs/use-with-claude/claude-usage.md)

## Connect
Add Trawl to any MCP client (Claude Desktop, Cursor, or any JSON-RPC 2.0 client) — one Streamable-HTTP endpoint plus a Bearer token:

- url: https://api.trawl.me/api/mcp
- header: Authorization: Bearer YOUR_TOKEN

Free token: npm install -g @trawlme/cli && trawl login. Pro: create a long-lived API key under Developers → API Keys.