BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Cloudflare Year in Review: AI Bots Crawl Aggressively, Post-Quantum Encryption Hits 50%, Go Doubles

Cloudflare Year in Review: AI Bots Crawl Aggressively, Post-Quantum Encryption Hits 50%, Go Doubles

Listen to this article -  0:00

Cloudflare has recently published the sixth edition of its Radar Year in Review. The results reveal 19% yearly growth in global internet traffic, Googlebot dominance, increasing crawl-to-refer ratios, and broad adoption of post-quantum encryption. Over 20% of automated API requests were made by Go-based clients, almost doubling adoption over the previous year.

Leveraging data from the widespread Cloudflare infrastructure, including anonymized query data for traffic to the 1.1.1.1 public DNS resolver, the yearly report analyzes the disruptions, advances, and metrics that defined the Internet in 2025. The review is organized into different sections (traffic, AI, adoption & usage, connectivity, security, and email security) and uses the same methodologies as previous years.

With global traffic growing 19% during the year and Google and Facebook still the most popular services, the report highlights significant growth for Starlink, up 2.3x year-over-year.

Using a Hilbert curve to visualize a sequence of IPv4 addresses in a two-dimensional pattern that keeps nearby IP addresses close together, the analysis finds that the Googlebot web crawler is the most popular. David Belson, head of data insight at Cloudflare, writes:

Googlebot was again responsible for the highest volume of request traffic to Cloudflare in 2025 as it crawled millions of Cloudflare customer sites for search indexing and AI training.

Furthermore, Googlebot was responsible for over 28% of traffic from verified bots, with Google AdsBot (used to monitor sites where Google ads are served), Google Image Proxy (used to retrieve and cache images embedded in email messages), and GoogleOther further increasing the dominance of the search company. OpenAI's GPTBot and Microsoft Bingbot follow at 7.5% and 6%, respectively.

The report reveals that AI platforms are aggressively crawling content without proportionally driving traffic back to source sites, with crawl-to-refer ratios increasing over 2024. Anthropic's ratio reached as high as 500000:1 and OpenAI peaked at 3700:1. Perplexity had the lowest crawl-to-refer ratios of the major AI platforms.

Jeremy Daly, director of research at CloudZero, in his newsletter, summarizes:

Cloudflare's excellent annual recap of a year filled with content hungry AI bots (Googlebot alone was responsible for 4.5% of all HTML requests and "user action" crawling jumped 15x), over half of human web traffic becoming post-quantum encrypted, and 174 major internet outages.

In the report, the hyperscaler acknowledges that Meta's llama-3-8b-instruct model was the most popular on Workers AI, its platform for running AI models directly at the network edge, with text generation being the most popular task type.

While requests using HTTP/3 and HTTP/2 both increased slightly in 2025, half of human-generated web traffic now uses post-quantum encryption, protecting against "harvest now, decrypt later" attacks, nearly doubling from 29% at the start of the year.

Post-Quantum Encryption
Source: Cloudflare blog

As in previous years, the team used Cloudflare Radar’s URL Scanner to identify the most popular technologies and services across the top 5000 domains and found that JavaScript-based libraries and frameworks remained integral tools for building websites. Belson adds:

jQuery is self-described as a fast, small, and feature-rich JavaScript library, and our scan found it on 8x as many sites as Slick, a JavaScript library used to display image carousels. React remained the top JavaScript framework used for building Web interfaces, found on twice as many scanned sites as Vue.js.

PHP, Node.js, and Java continued to be the most commonly used programming technologies, holding a clear lead over alternatives such as Ruby, Python, Perl, and C. In a popular discussion on Hacker News, many question the relative share of ASP.NET and C#, with user nic547 writing:

ASP.NET could mean a bunch of programming languages and I'm assuming that a ASP.NET Server doesn't disclose that. It's probably safe to guess mostly C#, but that requires a different metric.

Analyzing API-related requests, Cloudflare identified the top languages used to build API clients: 20% of automated API requests were from Go-based clients, a significant increase from Go’s 12% share in 2024. Python, Java, and Node.js followed as the most popular technologies.

API ClientLlibrary Popularity

Source: Cloudflare blog

While practitioners tend to focus on cloud outages, nearly half of the observed outages in 2025 were due to planned shutdowns intended to "prevent cheating on academic exams," with others related to protests and civil unrest, or cable cuts affecting both submarine and domestic fiber optic infrastructure.

As in previous years, the hyperscaler highlighted the increasing frequency and scale of hyper-volumetric network-layer attacks, defined as attacks that operate at Layers 3/4 and peak at more than one terabit per second or more than one billion packets per second. Chirag Goswami, founder of Cybernara, comments:

The Internet is having a midlife crisis. Bots are the new normal. DDoS attacks broke records. And one BGP oopsie can still knock out half the web. Cloudflare’s Radar Year in Review isn’t just stats, it’s a stress test report for how fragile, fast, and bot-infested the internet really is.

The Cloudflare Radar 2025 Year in Review microsite provides more detailed data, including trends by specific country and region.

About the Author

Rate this Article

Adoption
Style

BT