AI Bot Usage Distribution in the Top 1 Million Sites

Top In AI Bot Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
GPTBot Disallow	33,651	3.37
Common Crawl Bot Disallow	25,450	2.54
Anthropic Claude Bot Disallow	21,387	2.14
Alexa Bot Disallow	18,877	1.89
AmazonBot Disallow	16,443	1.64
Google Bard Disallow	15,940	1.59
Facebook Bot Disallow	14,223	1.42
ByteDance Bytespider Disallow	12,936	1.29
Perplexity	11,571	1.16
Cohere AI Disallow	10,774	1.08
Apple AI Bot Disallow	9,895	0.99
Webz Omgili Disallow	9,727	0.97
Diffbot Disallow	5,995	0.6
YOU Disallow	4,300	0.43

211,169 Detections

of AI Bot in the Top 1 Million Sites. Last updated 07th Mar 2025.

GPTBot Disallow is currently the most popular technology in this category.