Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	153,218	15.32
Ahrefs Bot Disallow	60,646	6.06
Googlebot Disallow	58,654	5.87
Yandex Bot Disallow	46,984	4.7
Google AdsBot Disallow	41,463	4.15
Some Robots Disallow	37,552	3.76
Baidu Bot Disallow	36,068	3.61
Semrush Bot Disallow	30,465	3.05
Majestic Bot Disallow	30,191	3.02
Microsoft BingBot Disallow	27,733	2.77
GPTBot Disallow	26,583	2.66
Google Media Partners Disallow	23,691	2.37
Yahoo Slurp Disallow	22,845	2.28
Alexa Bot Disallow	22,636	2.26
Common Crawl Bot Disallow	18,917	1.89
Sogou Bot Disallow	17,605	1.76
Soso Bot Disallow	14,864	1.49
Microsoft MSNBot Disallow	13,048	1.3
Anthropic Claude Bot Disallow	12,018	1.2
Google AdsBot Mobile Disallow	11,633	1.16
Google Bard Disallow	11,523	1.15
Google Image Bot Disallow	10,872	1.09
AppleBot Disallow	8,893	0.89
Exalead Bot Disallow	8,766	0.88
Facebook Bot Disallow	8,165	0.82
Google AdsBot Mobile Apps Disallow	7,658	0.77
Google News Bot Disallow	6,866	0.69
ByteDance Bytespider Disallow	6,483	0.65
DuckDuckGo Bot Disallow	5,632	0.56
Google Video Bot Disallow	5,352	0.54
Perplexity	5,289	0.53
Cohere AI Disallow	5,231	0.52
360 Bot Disallow	5,208	0.52
Webz Omgili Disallow	5,131	0.51
Apple AI Bot Disallow	4,910	0.49
Google Favicon Disallow	4,709	0.47
Google API Bot Disallow	4,610	0.46
Google DuplexWeb Disallow	4,508	0.45
BuiltWithBot Disallow	2,255	0.23
Diffbot Disallow	2,126	0.21
Pipl Disallow	1,772	0.18
YOU Disallow	1,449	0.14
Youdao Bot Disallow	950	0.1
Google FeedFetcher Disallow	135	0.01
Google Read Aloud Disallow	45	0
Google Web Light Disallow	17	0
Google BIG Bench Disallow	2	0