On July 2, 2026, Cloudflare announced it will automatically block mixed-use crawlers that both index websites for search and train AI models, giving website owners more control. Starting September 15, 2026, new domains will default to allow search crawlers but block training and agent bots on pages with ads, and multi-purpose crawlers will be blocked unless site owners opt in. The company is evolving its 2025 Pay Per Crawl program into a "Pay Per Use" model that compensates publishers when their content appears in AI chatbot answers. Cloudflare CEO Matthew Prince said these policies aim to rebalance the relationship between websites and AI companies and encourage companies to separate search, agent, and training crawlers.
