Cloudflare CEO Urges UK Regulator to Unbundle Google’s Search and AI Crawlers to Ensure Fair Competition
Cloudflare CEO Matthew Prince is urging the UK’s Competition and Markets Authority (CMA) to require Google to unbundle its search crawler from its AI data collection systems. Prince made the case during a visit to London, where he met with regulators to advocate for stronger rules governing how dominant tech firms like Google can operate in the AI space. The CMA recently granted Google special regulatory status due to its entrenched position in search and advertising, allowing the authority to scrutinize its activities beyond those markets—including AI Overviews, AI Mode, Discover, Top Stories, and the News tab. Prince argued that this designation is a crucial step toward ensuring fair competition. He emphasized that Cloudflare, while not an AI company or media publisher, is uniquely positioned to advise regulators because it serves as a network backbone for about 80% of AI companies. “We don’t have a dog directly in the fight,” Prince said at the Bloomberg Tech conference in London. “We’re not an AI company. But we’re this network that sits between them.” Prince highlighted a key issue: Google uses the same web crawler for both search and AI training, giving it a massive advantage. Unlike other AI firms, which must negotiate access or pay for content, Google can freely scrape the web under the guise of its long-standing search operations. “Google is saying, ‘We have an absolute God-given right to all of the content in the world, even if we don’t pay for it,’ Prince said. “And they’re using the same crawler for search and AI. If you want to opt out of one, you have to opt out of both.” That creates an impossible choice for publishers. Blocking Google’s crawler not only harms search visibility—potentially cutting off 20% of a media site’s traffic—but also disables Google’s ad safety systems, halting ad delivery across all platforms. “That’s just a non-starter,” Prince noted. This bundling, he argued, effectively grants Google a monopoly over AI training data, stifling competition. He believes the solution lies in separating the crawler from AI use, enabling thousands of AI companies to compete fairly by directly licensing content from publishers. Prince also shared internal data with the CMA showing how Google’s crawler infrastructure is far more extensive and efficient than those of competitors, making it nearly impossible for others to replicate the same scale. His concerns echo those of Neil Vogel, CEO of People, Inc., the largest U.S. publisher with over 40 media brands. Vogel recently called Google a “bad actor” and said media companies have no real choice but to allow AI scraping due to the bundled nature of the crawler. He confirmed that his company has adopted Cloudflare’s paid AI crawler model, which blocks non-paying bots and has led to active negotiations with major LLM providers.
