Cloudflare has introduced a new /crawl endpoint for its Browser Rendering service, now available in Open Beta. The company says developers can crawl an entire website with a single API call and export the results as HTML, Markdown, or structured JSON, giving AI teams a cleaner path for training datasets and RAG workflows.
One request starts a full-site crawl in async mode
The feature was announced on March 10. According to Cloudflare, the endpoint runs asynchronously: a developer submits a starting URL, receives a job ID, and the system then uses a headless browser in the background to discover and render pages across the site. Progress and results can be checked later through that same job ID. It is a simple workflow on the surface, but it bundles discovery, rendering, and extraction into one job.
Output options are a major part of the release. Alongside standard HTML, the API can return Markdown, which is widely used in large language model pipelines, and structured JSON powered by Workers AI. That cuts down on formatting work for teams building internal knowledge bases, preparing website content for retrieval systems, or assembling model training material.
Built as a compliant crawler, not a bypass tool
Cloudflare is positioning the endpoint as a transparent and rules-based crawler. The company says /crawl is a signed-agent and, by default, follows the target site’s robots.txt directives, including crawl-delay instructions. It also respects Cloudflare’s own AI Crawl Control policies.
Cloudflare also drew a hard line on access controls. The tool identifies itself as a bot and cannot bypass Cloudflare bot detection systems or CAPTCHA checks. That makes the scope clear: the endpoint is designed to collect content within site rules rather than help developers get around them.
Incremental crawling and static mode aim to lower cost
Cloudflare added several controls to make crawling more efficient. Incremental crawling supports modifiedSince and maxAge parameters, allowing developers to skip pages that have not changed or were fetched recently. Users can also set crawl depth, cap the number of pages, and include or exclude URL paths with wildcards.
For static sites that do not require JavaScript rendering, developers can set render: false to skip headless browser rendering and speed up collection. Cloudflare says the feature is available to both free and paid Cloudflare Workers users. That puts the tool within reach for teams tracking website updates, compiling research material, or maintaining enterprise AI knowledge bases.