Extract clean Markdown and JSON from websites with webclaw

webclaw turns web pages into cleaner Markdown, JSON, and LLM-ready context. It can run as a CLI, MCP server, REST API, or self-hosted service, making it useful for research, RAG ingestion, and read-only agent workflows that would otherwise receive noisy HTML.

What it can do

A single URL can be scraped into Markdown, while crawl and extraction modes handle a broader page set or a defined schema. Prebuilt binaries are available for Windows, macOS, and Linux. The project also provides an installer for connecting MCP-compatible clients.

Start with one public page and inspect the extracted headings, links, tables, code blocks, and source URL. Only then expand to a crawl. Set a narrow domain and path scope, cap page counts, and keep the source URL with every record.

Limits and data handling

Successful extraction does not grant republishing or training rights. Respect site terms, robots directives, copyright, privacy, and reasonable request rates. Do not use proxy or anti-blocking features to bypass authentication, paywalls, or explicit access controls.

Local extraction cannot handle every JavaScript application or bot defense. Hosted processing adds a privacy and cost tradeoff because requests and page content leave your environment. Automated cleanup can also remove important tables, footnotes, or interactive context, so consequential research still needs source-page verification.

Source and license

webclaw is maintained in the official 0xMassi/webclaw repository. The open-source CLI, MCP server, extraction engine, and self-hostable server use AGPL-3.0. At review time, the latest GitHub Release is v0.6.16. The hosted API is a separate service with its own operational and data-processing terms.