Shoal Research said in a new report that bots now account for 60% of HTML-content HTTP requests on Cloudflare’s network, compared with 38.2% for humans. The report cited earlier remarks from Cloudflare CEO Matthew Prince, who had predicted that bot traffic would surpass human traffic in 2027, and later said that growth in agent traffic had already pushed bots ahead of humans.
The report argued that AI applications are keeping users inside chat interfaces by summarizing content instead of sending them back to the original sites. It also said AI crawlers show a much higher crawl-to-return ratio than Googlebot. Over the past year, ClaudeBot reportedly fetched about 4,200 pages on average for every single return visit.
Shoal Research also pointed to a Pew study showing that AI summaries reduced click-through rates from 15% to 8%. In that setting, the report said, robots.txt and the long-standing model of trading content for traffic are becoming harder to sustain. Litigation and private licensing were described as temporary fixes. It grouped pay-per-request models into two camps: closed marketplaces with keys, and open protocols such as x402 and MPP that use HTTP 402 and stablecoins. The report added that most websites still either provide content for free or block access.
Shoal Research said in a report that bots now make up about 60% of HTTP requests for HTML content across Cloudflare’s network, while humans account for 38.2%.
The report referenced earlier comments from Cloudflare CEO Matthew Prince, who had predicted that bot traffic would surpass human traffic in 2027. It said Prince later announced that growth in agent traffic had already pushed bots above human traffic.
AI apps are changing how users reach content
According to the report, AI applications are keeping users inside chat interfaces and summarizing content there, rather than directing them back to the original websites.
The report also said AI crawlers have a far higher crawl-to-return ratio than Googlebot. Over the past year, ClaudeBot fetched about 4,200 pages on average for each return visit.
AI summaries are cutting click-through rates
A Pew study cited in the report showed click-through rates falling from 15% to 8% when AI summaries are present.
Shoal Research said robots.txt and the model of exchanging content for traffic are becoming difficult to sustain. It described litigation and private licensing as stopgap measures.
Two pay-per-request models
The report divided pay-per-request systems into two categories:
- closed markets that rely on keys;
- open protocols such as x402 and MPP that use HTTP 402 and stablecoins.
It added that most websites still either offer content for free or choose to block access.
This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan. Disclaimer:
The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.
Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.
