Major Publishers Lost 23% of Traffic After Blocking AI Bots, Though Smaller Sites May Face Different Tradeoffs

Welcome to AI Papers Explained, an experiment in using AI to help translate the latest AI research into plain language for journalists and technologists (we're getting meta). We're scanning for papers on arXiv, an open-access repository where researchers share preprints — papers that haven't yet gone through formal peer review. These summaries are AI-generated and lightly edited, and may contain errors or omissions.

Paper: The Impact of LLMs on Online News Consumption and Production
Authors: Hangcheng Zhao (Rutgers Business School) and Ron Berman (The Wharton School, University of Pennsylvania)
Published: December 2025

News publishers face an increasingly difficult choice about whether to block AI companies from crawling their sites.

New research quantifies the stakes of that decision for major news organizations. Large publishers who blocked AI bots from crawling their sites saw total website traffic drop by 23% — and that's not just from removing bot visits. Human traffic declined by 14% too.

The findings suggest that AI-powered search and summarization tools have already become significant referral traffic sources. For major publishers, blocking the bots appears to mean cutting yourself off from readers who discover your content through AI assistants.

The study also found that blocking effects vary by publisher size, with some mid-sized publishers experiencing traffic increases rather than declines, though these findings are more exploratory and the mechanisms aren't fully understood.

Who Did This Research and How

The study comes from business school researchers Hangcheng Zhao (Rutgers) and Ron Berman (Wharton) who specialize in digital platforms and algorithms.

The researchers combined multiple large-scale datasets:

• Traffic data: Comscore panel data tracking human visitor traffic to hundreds of news publishers, plus SimilarWeb data for verification. They focused on the top 500 news publishers by traffic, with the top 30 accounting for 69% of total traffic.
• Bot blocking behavior: Historical robots.txt files from the HTTP Archive to identify when publishers added "Disallow" rules to block AI crawlers from companies like OpenAI and Anthropic.
• Control group: Top 100 retail websites to isolate AI's effect on news specifically.

The analysis covers 2023 through 2024, capturing when AI-powered search tools moved from novelty to mainstream adoption.

What the Numbers Show

The study documents four key effects:

News traffic is declining. Traffic drops to news publishers became significant after August 2024, when news site traffic decreased by approximately 13.2% compared to retail websites.

Blocking AI bots reduces both total and human traffic for large publishers. Among the top 30 publishers who blocked AI crawlers, total traffic declined by 23%. Human traffic (measured separately by Comscore) declined by 14%, showing this wasn't just the mechanical removal of bot visits.

Some publishers reversed course. The negative effects were significant enough that some publishers removed their blocking rules after implementing them.

Effects appear to vary by publisher size. When researchers extended the analysis to a broader set of 500 publishers, they found more mixed results. Publishers with moderate traffic levels showed traffic increases after blocking, while the smallest publishers showed unclear effects. However, the study's primary focus and clearest evidence concerns the top 30 major publishers.

The Publisher's Dilemma

News organizations face difficult tradeoffs:

Allow AI crawling: Your content trains models and powers AI search tools that may answer queries without sending traffic to your site.

Block AI crawling: For major publishers, you lose significant traffic — not just from bots, but from humans who discover content through AI-powered search. You risk becoming invisible in the emerging AI-mediated information ecosystem.

For the large publishers that dominate news traffic, blocking AI bots was associated with losing nearly a quarter of their total traffic. The calculus may differ for smaller publishers, though the evidence here is less definitive.

Beyond Traffic: Jobs and Content

No evidence of newsroom job cuts yet. The share of editorial and content-production job listings actually increased over time, suggesting AI hasn't replaced core newsroom roles in the near term.

Content shifts to multimedia, not more text. Large publishers didn't increase text volume. Instead, they significantly increased rich content like video and interactive elements (68.1% increase) and ramped up advertising and targeting technologies (50.1% increase). This suggests publishers are creating experiences that AI tools can't easily replicate.

Important Caveats

Robots.txt compliance is imperfect. Research shows many AI bots don't actually check robots.txt files, yet traffic effects occurred anyway.

Correlation doesn't prove causation. The paper acknowledges that "blocking decisions may coincide with other time-varying publisher actions (paywall changes, SEO strategy, site redesigns or platform shocks)." Publishers who block may differ systematically from those who don't.

Unanswered questions:

• How do effects vary by news category (local vs. national, breaking vs. analysis)?
• What happened to publishers who reversed their blocking decisions?
• Why do some mid-sized publishers appear to benefit from blocking while large ones don't?
• What are the longer-term effects on newsroom employment?

The paper also doesn't address whether traffic from AI tools converts to sustainable revenue when those tools reduce the need to visit publisher sites.