Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	158,399	15.84
Ahrefs Bot Disallow	68,697	6.87
Googlebot Disallow	55,395	5.54
Google AdsBot Disallow	47,705	4.77
Yandex Bot Disallow	43,211	4.32
Some Robots Disallow	40,762	4.08
GPTBot Disallow	33,864	3.39
Semrush Bot Disallow	33,764	3.38
Majestic Bot Disallow	33,302	3.33
Baidu Bot Disallow	32,675	3.27
Common Crawl Bot Disallow	25,570	2.56
Microsoft BingBot Disallow	24,306	2.43
Anthropic Claude Bot Disallow	21,839	2.18
AppleBot Disallow	21,483	2.15
Sogou Bot Disallow	19,740	1.97
Google Media Partners Disallow	19,433	1.94
Alexa Bot Disallow	18,409	1.84
Yahoo Slurp Disallow	18,218	1.82
Google Bard Disallow	16,201	1.62
Facebook Bot Disallow	14,534	1.45
Microsoft MSNBot Disallow	13,369	1.34
ByteDance Bytespider Disallow	13,305	1.33
Google AdsBot Mobile Disallow	12,456	1.25
Perplexity	11,960	1.2
Cohere AI Disallow	11,037	1.1
Google Image Bot Disallow	10,927	1.09
Soso Bot Disallow	10,576	1.06
Apple AI Bot Disallow	10,206	1.02
Webz Omgili Disallow	9,952	1
Exalead Bot Disallow	9,705	0.97
Google AdsBot Mobile Apps Disallow	8,090	0.81
Google News Bot Disallow	6,902	0.69
Diffbot Disallow	6,343	0.63
360 Bot Disallow	6,243	0.62
DuckDuckGo Bot Disallow	6,225	0.62
Google Video Bot Disallow	5,470	0.55
Google Favicon Disallow	4,811	0.48
Google API Bot Disallow	4,738	0.47
Google DuplexWeb Disallow	4,617	0.46
YOU Disallow	4,589	0.46
BuiltWithBot Disallow	2,801	0.28
Pipl Disallow	2,546	0.25
Youdao Bot Disallow	911	0.09
LLMS Text	232	0.02
Google FeedFetcher Disallow	135	0.01
Google Read Aloud Disallow	57	0.01
Google Web Light Disallow	15	0
Google BIG Bench Disallow	5	0