Meta’s AI web crawlers generated a combined 9.1 billion requests in Q2 2026, according to a traffic report from security company DataDome, as the company’s search-indexing bot posted its fastest growth and independent developers complained about the strain on their servers.
Pieter Levels, an independent developer, wrote on X this week that Meta had been crawling all of his websites at “very very very high” volume. He said the traffic was heavy enough to trigger a load-average alert on one of his VPS instances. He also noted that Meta was hitting url2og, a screenshot service he built for his own sites, which suggested the crawler was not only collecting text.
DataDome says Meta accounted for more than half of AI agent traffic it handled in Q2
DataDome said its network processed 17.7 billion AI agent requests during the quarter, up 45% from the previous quarter. Of that total, Meta’s two crawlers accounted for 9.1 billion requests, more than half of the volume observed by the company. The report said Meta did not hold that level of dominance in Q1.
The two bots serve different functions.
- Meta-ExternalAgent is used for training. DataDome described it as behaving like a researcher that moves across the web reading material and feeding it into model training. Its quarterly request count rose from 3.1 billion to 5.3 billion, up 74% quarter over quarter.
- Meta-WebIndexer is used for search indexing. Its behavior is closer to that of a Google crawler, continuously reading pages and building an index so Meta AI can retrieve newer information when answering questions. Its quarterly request count jumped from 1.4 billion to 3.75 billion, a 163% increase.
In June 2026, Meta-WebIndexer’s monthly request volume exceeded Meta-ExternalAgent’s for the first time. The article says that marks a change in emphasis, from gathering data for model training to building an index that can support live lookups, and that the shift took place during this quarter.
Heavy crawling, little referral traffic
DataDome’s figures also showed that Meta’s crawlers sent little actual visitor traffic back to the sites they scraped. In the same report, ChatGPT captured 88% of AI referral traffic.
In other words, the company doing the most crawling was not the one sending the most traffic back.
The gap looked even wider in a smaller sample from tracking firm TrustMRR. Over a three-day measurement window, Meta made 67,390 requests, compared with 409 for OpenAI and 721 for Anthropic. Meta’s total was more than 160 times OpenAI’s during that period.
For site operators, those requests translate into direct costs. Bandwidth, CPU usage, and database queries all consume resources. Levels’ server alert is one example cited in the article. The complaint from publishers and developers is straightforward: a site may end up paying the infrastructure cost of serving a search engine without receiving the user traffic a search engine would typically return.
Meta has been building its own search index since 2024
The report says the current crawler activity did not begin as a short-term move. Earlier reporting from October 2024 said Meta had assigned a dedicated engineering team to build its own search database for Meta AI, after roughly eight months of work, with the goal of reducing dependence on Google and Microsoft Bing. The effort was led by senior engineering manager Xueyuan Su.
The article also says Meta-ExternalAgent has been operating on the web since the summer of 2024.
At the time, the search index was described as a way for Meta AI to produce its own news summaries and as a backup in case Meta’s relationships with outside search providers changed. Based on the newer traffic data, the article argues that what was once a backup now appears to be a core path.
On motive, the piece cites a private comment from one Meta employee who said the company wanted its own “Google” so it would not have to send AI search queries to Google and potentially feed Google’s own training efforts. The article notes that this came from a single anonymous source and was not confirmed by Meta, while adding that the claim moves in the same direction as the traffic pattern DataDome reported.
How Meta-WebIndexer differs from Meta-ExternalAgent
According to the article, Meta-ExternalAgent is a training crawler built to collect content at scale for model development. Meta-WebIndexer behaves more like a conventional search crawler, constantly building an index that Meta AI can use to answer fresher questions. In Q2, their request volumes were 5.3 billion and 3.75 billion, respectively, and on a monthly basis WebIndexer had already moved ahead in June.
Can websites block Meta’s AI crawlers?
The article says websites can use robots.txt to deny access to specific user agents, and Meta provides crawler documentation. But robots.txt is voluntary. In practice, site owners often need additional defenses such as a web application firewall, rate limiting, or services like Cloudflare to deal with high-frequency scraping.