How Cloudflare Is Making AI Pay for Content — and What It Means for Agent Email

How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?

For most of the web's history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person's behalf, requests a page, and leaves without engaging with any ads or subscriptions.

Cloudflare seeks to change this.

As you might be aware, Cloudflare sits between a large share of the world's websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402.

The Web's Old Bargain Is Breaking

For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit. The request itself stayed free, and this setup funded a large part of the Internet.

A growth in agent traffic is changing this dramatically. An agent is software that acts on a person's behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.

The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people. This means that request volume climbs while revenue stays flat.

To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned.

Cloudflare's Position as a Reverse Proxy

A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare's network first and passes through it on the way to the origin. The origin is the site's own server, the machine that ultimately holds the page or runs the API.

On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare's scale.

Classifying Traffic by Behavior, Not Just Labels

Before any rule can apply to an agent, the traffic has to be sorted by what it does. Cloudflare's taxonomy groups automated traffic by behavior rather than by the single label "AI". Three behaviors impact the key policy decisions:

Search covers behavior that builds an index of a site so an engine can answer questions about it later. This behavior is responsible for sending referral visitors back.

Agent covers behavior that acts in real time on a person's behalf, usually with a human waiting for the result. Think of it like an assistant fetching a page during a conversation.

Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back.

These three appear identical in a raw request log, yet they carry very different consequences for a site's business. A single crawler can also perform more than one of these behaviors, but this separation lets them record all of them. A crawler that both builds a search index and gathers training data is tracked as doing both, which lets a site owner reason about the full set of things that the crawler does on their pages.

From Blocking to Pay-Per-Crawl to Pay-Per-Use

Cloudflare's first solution was a simple one. A single control lets a site block automated AI traffic outright. This protected the content and left the earning model untouched. Blocking answers the question of access, yet it does not rebuild the revenue that the old bargain provided.

The second solution added a middle path. Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record. A site owner could keep a crawler out, let it through, or attach a price to its access, all from one setting. This turned the binary of block-or-allow into three options and gave content owners a way to earn from crawler access at network scale.

A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit. Counting crawls, then, counts something that only loosely tracks the value delivered.

So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point. This is because while pricing the outcome aligns payment with value more closely than pricing the fetch, it is also harder to measure.

Settling Identity, Permission, and Payment at the Edge

The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds.

Three main concerns are considered:

Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare's answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source. In practice, the operator signs its request with a private key and publishes the matching public key at a known location, and Cloudflare validates the signature at the edge. A valid signature stands in for a reliable identity, which replaces a guess with a verifiable claim.

Permission: This is expressed through the behavior classification and through the preferences a site sets about how its content may be used.

Payment: This comes last and is attached to the request through x402.

All three concerns resolve at the edge, so the origin receives a request only once identity, permission, and payment have been settled. The metering and settlement are taken away from the website's own servers. What stays with the site owner is the part that matters to them, which is their rules, their prices, and their revenue.

To summarize:

Identity answers who is making the request, through a signed and verified claim.

Permission answers whether this behavior is allowed on these pages, through the classification and content preferences.

Payment answers whether the caller has paid the stated price through the x402 exchange.

These pieces sit at different stages of maturity. Identity verification through Web Bot Auth is available today at the edge, while the Monetization Gateway that brings the payment piece together is open as a waitlist rather than a shipped product.

The key takeaway is that when one component sits in the middle of a flow, the concerns that are common across every request are collected at that point. Authentication, authorization, and billing consolidating at a gateway is the same pattern that appears in service meshes and middleware.

How x402 Works: Payment Over HTTP

The x402 protocol makes it possible to pay over HTTP. It takes its name from a status code that has been part of the HTTP standard for a long time. The code is 402, and it means Payment Required. Sites behind Cloudflare already send more than a billion of these responses on an average day, which shows how often a machine requests something priced and receives a message that a payment is due.

The exchange runs through a short sequence that can be compared to a small state machine:

A client requests a resource that sits behind a price.

Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay.

The client re-sends the same request with proof of payment attached.

A facilitator verifies the payment, and the server returns the resource.

Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402

Content Independence Day, one year on: building the business model for the agentic Internet

Making AI search smarter

Your site, your rules: new AI traffic options for all customers

Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Launching the x402 Foundation with Coinbase, and support for x402 transactions

Web Bot Auth

Forget IPs: using cryptography to verify bot and agent traffic

This article is based on the original ByteByteGo post "How Cloudflare Is Making AI Pay for Content" with added Mermail commentary. Original: Blog.ByteByteGo.com

Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment. This property is the one that matches an anonymous agent passing through once, since it removes the signup step that per-seat licensing and API keys always required.

The negotiation process is handled inside ordinary requests and responses, with a redirect to a checkout page absent and a separate payment API absent. Nothing was added at the protocol level, since the 402 code has been part of HTTP for decades. What changed is that a settlement angle now exists that makes collecting a fraction of a cent practical.

Costs, Trade-offs, and Open Questions

Resolving identity, permission, and payment at the edge means one proxy performs those functions for a large share of the web at once. Cloudflare presents that responsibility as an advantage for settling everything inside one request. But it can also be seen as a potential risk when so much runs through a single provider.

Cloudflare also states several limits plainly:

Trust that travels with a request may reach only the traffic that can afford to be identifiable, and small or privacy-sensitive sources of traffic need other building blocks, such as private rate limiting.

Usage-based payment helps a site that already has demand, and does little for a small site whose real difficulty is discoverability rather than monetization, which leaves that site weighing visibility against giving content away.

Collecting a payment through this exchange depends on callers built to recognize and honor the 402 response. Therefore, the revenue depends on adoption within the ecosystem.

Pricing an outcome rather than a crawl aligns payment with value, but it is also harder to measure and verify. This is the reason Cloudflare frames the shift to Pay-Per-Use as an experiment.

None of these costs undoes the shift. They simply mark the current trade-offs that website owners should consider before adopting it.

The Bigger Picture: Moving Settlement Inside the Request

The web is moving value settlement from after the request to inside it. For most of the web's history, a request was served free, and value was settled later through human attention. However, agent traffic, which now comprises the majority of requests, leaves that later settlement with nowhere to land.

Cloudflare's response is to resolve four things from its position as a reverse proxy: seeing what a request is through classification, verifying who sent it through Web Bot Auth, enforcing the site's rules, and settling payment through the x402 exchange, all before the origin responds.

Identity verification lives at the edge today, and the payment gateway opens as a waitlist. The open questions around concentration, reach, adoption, and how to price an outcome are worth tracking as the model develops.

What This Means for Agent Email — Mermail's Perspective

Cloudflare's thesis — that the agentic web needs identity, permission, and payment settled at the edge — extends far beyond web content. It applies just as forcefully to email, which is the original open protocol for machine-to-machine communication.

This is exactly the problem space Mermail is building for. Mermail is a privacy-first email inbox API designed for AI agents. Instead of forcing agents through IMAP, SMTP, or brittle scraping layers, Mermail gives agents a native API to receive, read, and act on email — with full control over identity and access.

Think about the parallel. Just as Cloudflare sits between the web and its callers to verify who's asking and settle payment before the origin server lifts a finger, Mermail sits between the inbox and the agent to verify that the agent has the right to read a message, route it, or act on it. The same three concerns — identity, permission, and payment — apply to agent email, and they're solved at the same edge layer.

Here's why this matters:

Agent email needs verified identity. When an AI agent reads your inbox, you need cryptographic assurance that it's your agent, not a malicious actor. Mermail's user-owned key model ensures that every agent accessing an inbox is cryptographically verified — much like Web Bot Auth verifies crawlers at the edge.

Agent email needs permission scoping. Not every agent should read every email. Mermail lets you define fine-grained routing rules: this agent gets transactional emails, that agent gets newsletters, and no agent sees your password resets unless you say so. This is the email equivalent of Cloudflare's behavior classification — sorting traffic by what it's allowed to do.

Agent email needs settlement. If an agent sends requests on your behalf that trigger paid services — booking a flight, ordering supplies, renewing a subscription — that transaction chain needs to settle somewhere. An inbox API that understands payment context from the start is positioned to be that settlement layer.

The pattern is the same whether you're serving web pages or processing email: move identity, permission, and payment to the edge, settle them before the real work begins, and let the origin focus on what it does best. Cloudflare proved this model for web content. Mermail is proving it for the inbox.

The agentic internet isn't just about scraping web pages — it's about agents acting on your behalf across every protocol, including email. And when they do, they'll need infrastructure that handles identity, permission, and value exchange at the edge. That's what we're building at Mermail.

References