From Googlebot to GPTBot: who’s crawling your site in 2025

This report analyzes web crawling trends from May 2024 to May 2025, showing overall AI and search crawler traffic growth (+18%) and a reshaping of the ecosystem: Googlebot expanded to 50% share (+96% in requests), OpenAI’s GPTBot surged to 7.7% (+305%), while Bytespider and ClaudeBot declined. It details shifting rankings across the top 20 crawlers, seasonal patterns, robots.txt directives targeting AI bots (with GPTBot most frequently blocked and also most explicitly allowed), and a broader move toward enforceable controls like WAFs as sites navigate content rights, privacy, and infrastructure impacts.