Search drives most experiences on the web. It’s about how we do things and how almost everything on the web is found – the creators, the retailers, the answer to whatever you just typed into a field. For nearly 30 years, this journey of discovery involved one simple transaction: let a search engine crawl your content and it will send you visitors. They turned those visitors into a business – through ads, subscriptions, or just the audience itself. Being discoverable and getting paid were the same thing. A year ago, on the first Content Independence Day, we drew a line to defend this agreement in the age of AI. But a line in the sand was only a first step. Since then, the prevalence of AI search in consumers’ lives has only increased, with more than 50% of online traffic being non-human in nature. The threat is no longer a handful of training crawlers that you can block; Search itself is being rebuilt around AI answers.
Today’s response engines read your page and give the user a summary, eliminating the need for the visit – and the revenue that comes from it. We see it firsthand, and independent research confirms it: A 2025 Pew Research Center study found that when Google displayed an AI summary, users clicked on a traditional search results link only 8% of the time (about half as often as without a summary), and only 1% of the time they clicked a link within the summary. This puts our customers in a quandary: choose against AI and be hard to find, or choose it and offer users significant added value while the return becomes less and less. Our customers want to be found and compensated for the value they provide, and right now they are forced to choose.
Today we announced new bot options to help our customers better control who can access their website and what they can do with it. But blocking was just the first step: saying “no” protects content without rebuilding the business models that support it. So it’s time to start building the Internet’s new economic model, starting with search.
Transparency and control are the foundation, but more is needed. In 2025, we laid the groundwork with a set of responsible AI bot principles: Bots should be transparent about who they are and what they do, respect the decisions of site owners, and act in good faith. Our tools keep bots at this level. But enforcing good bot behavior doesn’t make AI search better for the people who rely on it, and it doesn’t return a dollar to the creator whose work made the answer possible. We can do more than help the web say “no.” We can help rebuild what it says “yes” to.
That’s why today we’re announcing two initiatives that move from defense to offense and begin to bring both halves of this old bargain back together.
Make AI Search Smarter: From Using the signals we see across our global network, such as what is current, what is high quality, and what has actually changed, we can help response engines show the most relevant content and reduce unwanted crawling. Searchers get better answers while costs go down for both AI companies and website owners when web pages are only recrawled when they have changed.
Pay creators for the value they provide: If your work is used to answer a question, you should be rewarded rather than just scratched for free. And you should be able to see what is being used and what people are asking. This should be a real source of income and an incentive to continue producing original content worth finding.
Today we’re launching a research program to make AI search smarter and stop our customers footing the bill for crawls that don’t produce anything new.
More than 20% of the web is behind Cloudflare’s network, giving us a unique perspective. We can see which sites have truly changed and which people and agents are flocking to. As part of this program, we will explore the use of signals that our customers want to share about the freshness of their content, and we will combine these with our own insights into traffic flows, both human and bot. For response engines, this is a signpost to high-quality content. For our customers, it provides an overview of what users are actually asking and how their content appears in the AI results. The goal is to measure two things: how much these signals help answer search engines display more current and higher quality content, and how much unnecessary crawling it avoids.
The second benefit of avoiding unnecessary crawling is bigger than it sounds. Cloudflare data suggests that more than 50% of crawl traffic is used by good bots to refetch pages that haven’t changed – and that number is likely to increase as crawl volume increases. A signal that simply says, “Nothing has changed here” will cause a crawler to skip the trip. This saves the answer machine’s computing power. More importantly, it saves website owners from having to process and pay for requests they don’t need.
The program is inherently neutral: our goal is to make it work for any answer machine willing to play fairly. It is limited to search. We do not share any content and nothing is used to train Foundation models. We plan to publish our findings, including benefits for site owners such as better content discoverability and reduced server load. We plan to make the feature generally available later this year and reduce unnecessary crawling on our network.
Table of Contents
From pay-per-crawl to pay-per-use
Last year we introduced Pay Per Crawl to allow publishers to charge AI companies to crawl their content. It was a real start, but crawling is a rough measure of value. A single page may be crawled once and then cited in thousands of answers, or it may be crawled again and again and never used. Creators want to be paid fairly for the value they provide.
So we start converting Pay Per Crawl to Pay Per Use. We’re running experiments with top AI companies like Ceramic.ai and You.com, and the deal is straightforward: companies can bring their payment models and easily scale them to content owners across the Cloudflare network.
Ceramic has developed a so-called “pay-per-query” model so that publishers who choose to do so can be paid when their content appears in Ceramic’s search results. This means that payment is based on the value the work delivers, not how often a crawler retrieves it.
“To scale the future of AI search, we need a partner with broad reach and a shared commitment to transparency and fair compensation,” he says Anna Patterson, Founder and CEO of Ceramic.ai. “Cloudflare allows us to easily and programmatically scale our operations. By integrating our pay-per-query model into their network, we ensure that millions of content owners can seamlessly sign up for compensation every time their content appears in our search results.”
In addition to compensation, content owners who participate in the Cloudflare/Ceramic program will unlock new reports to help with response engine optimization (AEO). Customers can finally see the top search queries that lead to their content appearing in search results, the specific webpage and snippet, their average position in search results rankings, and more. This is the first of many products we will launch to make discoverability easier for our customers.
This is just a new approach. Another benefit comes from You.com: Agents can pay on-demand for specific premium content they need without making an upfront commitment. New payment models from AI providers are being tested (e.g. Pay per Query, Pay per Result, etc.) and we have the infrastructure to support them all.
Let’s be honest that this is an experiment. There is a lot to learn, including how exactly this works at the scale of the Internet. We will work this out over time with our partners and customers and share our findings. But the goal is clear: AI search companies get more timely, informed answers, and the customers whose work makes the answers possible get paid to help. Cloudflare’s job in all of this is to provide the infrastructure layer that makes this market thrive.
We believe this is more in line with the future economic development of search. The old human web optimized search to save time – with excerpts, ten blue links and one click. The agent internet is different: an agent can read quickly and search continuously. The search becomes something an agent performs dozens of times to answer a single question that is closer to a utility than a destination. In this world, crawling or clicking doesn’t matter. It’s the result. Pricing the result and paying the people who made it possible are the reasons the web continues to thrive.
The headline we want to earn
A year ago, on Content Independence Day, the default headline was “No”: AI can’t crawl without compensation. This year, our focus is to give our users more products and controls to help them say “yes” and reap more benefits.
Today’s announcements are just the beginning. Cloudflare’s research project aims to find out whether our signals produce better results with less crawling. Pay Per Use is a promising direction that we will experiment with with partners who believe that content creators deserve fair compensation for their work. This is how the last 30 years of the Internet have been structured: Someone runs the pilot that turns “The model is broken” into “Here is the new model,” one experiment at a time. We believe there is value for our customers in being discoverable in this new age of agents and optimizing their content for maximum discoverability. But they should be able to do this without giving away their most valuable creative resources for free.
The Internet is changing and so are the business models it relies on. The old internet was open, neutral and worthy of input. We have the rare chance that this will stay that way and that the business models will be built that will continue to finance this in the future. Smarter answers for the people and agents asking the questions. A fair deal for the people whose skills, creativity and commitment make the answers worthwhile. This is how we pursue Cloudflare’s mission: to help build a better Internet.
Happy Content Independence Day!
Building on the open, agent-ready web? If you would like to learn more about the Ceramic and You programs, please fill out the form this form. If you want to build a response engine and crawl smarter, we’d love to hear from you too: [email protected].
https://blog.cloudflare.com/making-ai-search-smarter/
