AI Bot Usage Distribution in the Top 1 Million Sites

Top In AI Bot Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
GPTBot Disallow	33,864	3.39
Common Crawl Bot Disallow	25,570	2.56
Anthropic Claude Bot Disallow	21,839	2.18
Alexa Bot Disallow	18,409	1.84
AmazonBot Disallow	16,774	1.68
Google Bard Disallow	16,201	1.62
Facebook Bot Disallow	14,534	1.45
ByteDance Bytespider Disallow	13,305	1.33
Perplexity	11,960	1.2
Cohere AI Disallow	11,037	1.1
Apple AI Bot Disallow	10,206	1.02
Webz Omgili Disallow	9,952	1
Diffbot Disallow	6,343	0.63
YOU Disallow	4,589	0.46

214,583 Detections

of AI Bot in the Top 1 Million Sites. Last updated 21st Apr 2025.

GPTBot Disallow is currently the most popular technology in this category.