Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	3,458	34.58
GPTBot Disallow	1,387	13.87
Some Robots Disallow	1,289	12.89
Googlebot Disallow	1,236	12.36
Common Crawl Bot Disallow	1,024	10.24
Ahrefs Bot Disallow	930	9.3
Anthropic Claude Bot Disallow	843	8.43
Google AdsBot Disallow	780	7.8
AppleBot Disallow	776	7.76
Yandex Bot Disallow	694	6.94
Semrush Bot Disallow	688	6.88
Google Bard Disallow	668	6.68
Facebook Bot Disallow	609	6.09
ByteDance Bytespider Disallow	609	6.09
Perplexity	590	5.9
Webz Omgili Disallow	565	5.65
Apple AI Bot Disallow	561	5.61
Baidu Bot Disallow	543	5.43
Cohere AI Disallow	524	5.24
Majestic Bot Disallow	516	5.16
Microsoft BingBot Disallow	499	4.99
Diffbot Disallow	385	3.85
Alexa Bot Disallow	381	3.81
Yahoo Slurp Disallow	292	2.92
Microsoft MSNBot Disallow	278	2.78
Google Media Partners Disallow	269	2.69
Google AdsBot Mobile Disallow	241	2.41
Google Image Bot Disallow	229	2.29
Sogou Bot Disallow	189	1.89
Exalead Bot Disallow	182	1.82
Google News Bot Disallow	172	1.72
Google AdsBot Mobile Apps Disallow	153	1.53
Soso Bot Disallow	130	1.3
YOU Disallow	128	1.28
360 Bot Disallow	113	1.13
DuckDuckGo Bot Disallow	113	1.13
Google Video Bot Disallow	96	0.96
Google API Bot Disallow	62	0.62
Pipl Disallow	61	0.61
Google Favicon Disallow	59	0.59
Google DuplexWeb Disallow	54	0.54
BuiltWithBot Disallow	39	0.39
Youdao Bot Disallow	27	0.27
Google FeedFetcher Disallow	1	0.01