Meta’s AI crawlers sent 9.1 billion requests in Q2 as WebIndexer surged and developers reported server strain
Meta’s web crawling operation expanded sharply in the second quarter of 2026, according to DataDome, with its two AI crawlers generating a combined 9.1 billion requests across the security firm’s network. The jump put Meta ahead of other AI agents tracked in the report and marked a notable shift in how the company appears to be gathering web data. One crawler, Meta-ExternalAgent, is used for model training and rose from 3.1 billion to 5.3 billion requests quarter over quarter. The other, Meta-WebIndexer, is tied to search indexing and climbed from 1.4 billion to 3.75 billion requests, a 163% increase. In June 2026, WebIndexer’s monthly volume surpassed ExternalAgent for the first time, suggesting Meta’s focus has moved toward building a live index rather than only collecting training data. Independent developer Pieter Levels said on X that Meta had been hitting all of his websites with “very very very high” levels of crawling, enough to trigger a load-average alert on one VPS. He also said Meta was requesting his url2og screenshot service, indicating that the crawler activity was not limited to text pages. DataDome said Meta’s crawlers drove little real traffic back to the sites they scraped, while ChatGPT accounted for 88% of AI referral traffic in the same report. The article also points to earlier reporting that Meta has been building its own search database since 2024 in an effort to reduce reliance on Google and Microsoft Bing.








