Creative Commons has announced tentative support for pay-to-crawl systems, a technology designed to automate compensation for website content when accessed by artificial intelligence webcrawlers. This move follows the nonprofit’s earlier initiative to create a legal and technical framework for responsible data sharing between data owners and AI developers. Creative Commons, best known for its open licensing model that empowers creators to share their work while retaining copyright, has long advocated for a more equitable digital ecosystem. In July, it unveiled a plan to help standardize how datasets are shared between companies and AI providers. Now, it is cautiously backing pay-to-crawl systems, which would require AI bots to pay a fee each time they scrape a website to collect data for training or updating models. The nonprofit stated that, if implemented responsibly, pay-to-crawl could help sustain content creation and preserve public access to information. “It could represent a way for websites to sustain the creation and sharing of their content, and manage substitutive uses, keeping content publicly accessible where it might otherwise not be shared or would disappear behind even more restrictive paywalls,” a blog post explained. Traditionally, websites allowed search engine crawlers to freely index their content in exchange for increased visibility and traffic. But with the rise of AI chatbots, that dynamic has shifted. Users now often receive answers directly from AI systems without clicking through to the original source—leading to a steep decline in referral traffic for publishers, particularly smaller ones. Pay-to-crawl offers a potential solution. It could help publishers recoup lost revenue and level the playing field, especially for those without the leverage to negotiate exclusive deals with major AI firms. While large-scale agreements have already been struck—such as between OpenAI and Condé Nast, Perplexity and Gannett, Amazon and The New York Times, and Meta and various media companies—smaller publishers have been left behind. However, Creative Commons issued several important caveats. It warned that pay-to-crawl systems could concentrate power in the hands of a few tech companies, potentially blocking access for researchers, educators, cultural institutions, and other public-interest actors. To address this, the organization laid out a set of principles for responsible implementation: pay-to-crawl should not be the default for all websites, should avoid blanket restrictions, and should allow for throttling rather than outright blocking. It also stressed the need for open, interoperable systems built on standardized components. Cloudflare is a leading proponent of pay-to-crawl, but it’s not alone. Microsoft is developing an AI marketplace for publishers, and startups like ProRata.ai and TollBit are entering the space. Another initiative, the RSL Collective, has introduced the Really Simple Licensing (RSL) standard, which defines what parts of a website AI crawlers can access without blocking them entirely. RSL has gained support from major tech firms including Cloudflare, Akamai, and Fastly, as well as media organizations like Yahoo, Ziff Davis, and O’Reilly Media. Creative Commons has also endorsed RSL, along with its broader CC Signals project—a suite of tools and technologies aimed at supporting transparency and fairness in AI data use. The nonprofit continues to advocate for a balanced approach that protects both creators and the public good.