The Pay Per Crawl Revolution: How Publishers Can Monetize AI Traffic

The era of free content scraping from the web is ending. Here's how new platforms are turning AI bots from non-paying visitors into real customers.

The Problem: When AI Breaks the Deal with Publishers.

The Google Model: A Balance That Worked

For over twenty years, the web has operated on an unwritten pact: Google and other search engines indexed publishers' content for free, showed snippets and summaries on their pages, but in exchange sent massive traffic back to the original sites. Publishers accepted this "use" of their content because they received visibility, readers and, as a result, advertising revenue.

Google News has perfected this model: it aggregates news from thousands of sources, shows headlines and short excerpts, but each click takes the user to the original site. A win-win balance that has sustained the digital news ecosystem.

AI Breaks the Balance.

Now imagine running a news site and discovering that every day thousands of AI bots scan your articles, use the content to power their models and serve complete answers to users, without ever sending traffic back to your site. AI that doesn't generate links, and doesn't generate clicks: it answers directly. Welcome to the reality of 2025.

According to Cloudflare data, by mid-2025 80% of AI crawling activity is destined for model training, while referrals to publishers (especially from Google) are dropping drastically. The model that worked for twenty years is crumbling: Google itself, with its AI Overviews, is sending less and less traffic to original sites. The "crawl-to-refer" ratio shows shocking numbers: Anthropic has a ratio of 38,000 crawls for every visitor sent to the site, while OpenAI reaches 1,700:1.

But what would happen if bots could become paying customers instead of simple free consumers of content? It's time for a new pact: if AI can no longer guarantee return traffic the way Google did, it must at least pay for access to content.

The Solution: Pay Per Crawl and AI Micropayments

Cloudflare Opens the Dances

In July 2025, Cloudflare announced it had become the first internet infrastructure company to block AI crawlers by default, simultaneously introducing the "Pay Per Crawl" system. The novelty? Every new domain registering with Cloudflare is now explicitly asked whether it wants to allow AI crawlers to access its content.

But the real game-changer is the marketplace where publishers can request compensation from AI companies every time one of their pages is crawled. No longer just "allow" or "block", but a third option: "charge".

TollBit: The Pioneer of Micropayment AI

Even before Cloudflare, TollBit had started building this infrastructure. The startup, which recently raised $24 million in Series A, has created a platform where AI bots can pay websites directly to use their content.

TIME and Adweek are among the first customers experimenting with this model, discovering that AI represents a new customer category with specific needs.

The Numbers: How Much Can You Really Make With It?

TollBit's Projections

According to a detailed analysis, TollBit could already be generating around $71 million a year for its publishing partners. Here's how:

• Exponential traffic growth: AI bot traffic to TollBit client sites has doubled every quarter
• Q4 2024: 2.75 million daily bot accesses
• Q1 2025: 6.5 million accesses
• Q2 2025 projection: 13 million accesses

Using a typical CPM of $15 for content sites, this translates to $195,000 per day in potential revenue for the TollBit ecosystem, or over $71 million annually.

The Pricing Model

TollBit suggests two levels of pricing:

• Summarization rate: For bots that want to summarize content
• Syndication rate: Much higher, for bots that want to display the full article

The difference can be substantial, making this a particularly attractive model for premium and specialized content.

Actual Data from the Field

A TollBit analysis of 160 websites revealed that AI companies crawled an average of 2 million times in Q4 2024. Each page was crawled 7 times on average.

Toshit Panigrahi, co-founder of TollBit, explains: "The bot traffic generated by these AI platforms is nearly equivalent to the bot traffic from search engines that have existed for 20 years, which is incredible."

The Evolving Ecosystem

DataDome: Protection and Monetization

DataDome has formed partnerships with both TollBit and other platforms to offer an integrated solution: real-time protection against malicious bots and monetization of compliant ones.

Their data shows that AI traffic has tripled in just six months, going from 2.6% to 8.2% of all verified bot traffic.

Skyfire: The AI Payments Network

Skyfire represents a different approach, building an entire payment network for autonomous AI agents. With $8.5 million raised and partnerships with companies like DataDome, Skyfire aims to become the standard payment infrastructure for the AI economy.

The Impact for Publishers: Opportunities and Challenges

The Concrete Benefits

1. New revenue source: Diversification beyond traditional advertising
2. Granular control: Ability to choose which bots to grant access to and at what price
3. Full visibility: Detailed dashboards on who accesses content and how often
4. Direct compensation: Immediate payments without advertising intermediaries

The Realistic Challenges

1. Limited adoption: Not all AI companies will agree to pay
2. Complex pricing: Difficult to determine the correct value per content type
3. Enforcement: Many bots will continue to ignore the rules (from 3.3% in 2024 to 13% in March 2025)
4. Scale required: Micropayments only become significant at high volumes

Realistic Forecasts for Publishers

The Threshold of Convenience: Blocking Doesn't Always Pay Off

Before analyzing the numbers, it's important to point out that Pay Per Crawl isn't worthwhile for everyone. For sites with limited traffic, allowing free access to AI bots can be strategically more advantageous: it increases content discoverability within AI systems, builds relationships with emerging platforms, and positions the site as "AI-friendly" ahead of future opportunities.

Direct monetization becomes attractive only when the volumes justify the management complexity and implementation costs.

For Small Sites (10K visits/month)

• Estimated bot traffic: 500-1,000 AI accesses/month
• Potential revenue: $5-50/month (depending on CPM and acceptance rate)
• Impact: Marginal, often better to remain open for visibility
• Recommendation: Monitor and evaluate strategic free access

For Medium Sites (100K visits/month)

• Estimated bot traffic: 5,000-15,000 AI accesses/month
• Potential revenue: $75-750/month
• Impact: Threshold where it starts to make sense to consider monetization
• Recommendation: Experiment with paid premium content

For Large Publishers (1M+ visits/month)

• Estimated bot traffic: 50,000-200,000+ AI accesses/month
• Potential revenue: $750-10,000+/month
• Impact: Potential substantial new business line
• Recommendation: Immediate implementation recommended

The Future of Content Licensing

Prashanth Chandrasekar, CEO of Stack Overflow, sums up the situation perfectly: "Community platforms that power LLMs should be compensated for their contributions so they can reinvest in their communities."

With AI-to-AI commerce projected to reach $46 billion over the next three years, publishers who position themselves early in this ecosystem could find themselves with a significant competitive advantage.

Conclusion: Is it the Time to Act?

"Pay Per Crawl" is no longer science fiction, but operational reality. For publishers, the question is not whether this model will take hold, but if and when they'll be able to adapt to capture its value.

The uncomfortable truth: not everyone should monetize right away. For sites with limited traffic (<50K visits/month), keeping open access for AI bots can be strategically more advantageous, building visibility and relationships for future opportunities. The convenience threshold for direct monetization is generally reached with significant traffic volumes.

Projections show attractive potential revenues for medium and large publishers, especially for quality content and substantial traffic. However, success will depend on the ability to:

1. Assess your own convenience threshold (volumes vs. management complexity)
2. Choose the right strategy (open access, total block, or monetization)
3. Choose the right platform (Cloudflare, TollBit, or integrated solutions)
4. Set competitive but profitable pricing
5. Maintain content quality that justifies the micropayments
6. Monitor and optimize the approach constantly

For publishers with substantial traffic willing to experiment, 2025 could be the year AI stops being just a cost and finally becomes a business opportunity. For everyone else, it may be the time to build the foundations to capture this opportunity in the future.

Main sources:

• Cloudflare Blog: AI Crawler Control
• TollBit Series A Announcement
• MIT Technology Review: Cloudflare AI Blocking
• DataDome Partnership Announcement
• Skyfire TechCrunch Coverage