People CEO Accuses Google of Being a 'Bad Actor' for Using Same Crawler to Steal Content for AI
Neil Vogel, CEO of People, Inc.—the largest digital and print publisher in the U.S., which owns brands like People, Food & Wine, Travel + Leisure, Better Homes & Gardens, Real Simple, Southern Living, and Allrecipes—has accused Google of acting as a “bad actor” by using the same web crawler to both index content for search and harvest it to train its AI systems. Speaking at the Fortune Brainstorm Tech conference, Vogel argued that Google’s lack of separate crawlers for search and AI gives it an unfair advantage, allowing it to use publishers’ content to compete directly with them. Vogel noted that three years ago, Google Search drove about 65% of People Inc.’s web traffic, a figure that has since fallen to the “high 20s.” In a previous interview, he shared that Google once accounted for as much as 90% of the company’s traffic from the open web. Despite the decline, Vogel emphasized that People Inc. has grown its audience and revenue, and he’s not complaining about the loss of traffic. However, he stressed that it’s unethical for Google to use publisher content to fuel its AI products while still relying on those same publishers for search traffic. He called on publishers to gain more leverage in the AI era by blocking AI crawlers—automated bots that scan websites to train large language models. People Inc. has adopted Cloudflare’s new solution to block non-paying AI crawlers, which has prompted several large LLM providers to reach out with potential content deals. Vogel said the company is “much further along” than before in negotiating such agreements, though no deals have been finalized. However, he pointed out that Google’s crawler cannot be blocked without risking the loss of vital search traffic. Since Google uses one crawler for both purposes, blocking it would hurt the publisher’s visibility in search results. “They know this, and they’re not splitting their crawler. So they are an intentional bad actor here,” Vogel said. Janice Min, CEO and editor-in-chief of Ankler Media, echoed these concerns, labeling Big Tech companies like Google and Meta as long-time “content kleptomaniacs.” She said her company blocks AI crawlers and sees no benefit in partnering with AI firms at this stage. Matthew Prince, CEO of Cloudflare, which developed the crawler-blocking technology, said he believes the landscape will shift. He predicted that by this time next year, Google will likely begin paying publishers to use their content in AI models. Prince also questioned the effectiveness of relying on outdated copyright laws, noting that AI training often involves creating derivative works—something historically protected under fair use. He pointed to Anthropic’s $1.5 billion settlement with book publishers as evidence that courts may view AI training as a legitimate use under current legal interpretations. Prince also blamed Google for shaping the digital media ecosystem in ways that prioritized traffic over content quality, contributing to click-driven journalism at outlets like BuzzFeed. Still, he acknowledged that Google faces intense internal pressure and competition, making its current position difficult to sustain.
