Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	163,669	16.37
Ahrefs Bot Disallow	69,476	6.95
Googlebot Disallow	57,398	5.74
Google AdsBot Disallow	48,482	4.85
Yandex Bot Disallow	44,307	4.43
Some Robots Disallow	41,977	4.2
Semrush Bot Disallow	33,897	3.39
Baidu Bot Disallow	33,893	3.39
GPTBot Disallow	33,651	3.37
Majestic Bot Disallow	33,459	3.35
Common Crawl Bot Disallow	25,450	2.54
Microsoft BingBot Disallow	24,860	2.49
Anthropic Claude Bot Disallow	21,387	2.14
AppleBot Disallow	20,992	2.1
Google Media Partners Disallow	20,621	2.06
Sogou Bot Disallow	19,904	1.99
Alexa Bot Disallow	18,877	1.89
Yahoo Slurp Disallow	18,858	1.89
Google Bard Disallow	15,940	1.59
Facebook Bot Disallow	14,223	1.42
Microsoft MSNBot Disallow	13,762	1.38
ByteDance Bytespider Disallow	12,936	1.29
Google AdsBot Mobile Disallow	12,931	1.29
Perplexity	11,571	1.16
Google Image Bot Disallow	11,292	1.13
Soso Bot Disallow	10,895	1.09
Cohere AI Disallow	10,774	1.08
Apple AI Bot Disallow	9,895	0.99
Exalead Bot Disallow	9,759	0.98
Webz Omgili Disallow	9,727	0.97
Google AdsBot Mobile Apps Disallow	8,442	0.84
Google News Bot Disallow	7,193	0.72
DuckDuckGo Bot Disallow	6,461	0.65
360 Bot Disallow	6,297	0.63
Diffbot Disallow	5,995	0.6
Google Video Bot Disallow	5,759	0.58
Google Favicon Disallow	5,115	0.51
Google API Bot Disallow	5,022	0.5
Google DuplexWeb Disallow	4,904	0.49
YOU Disallow	4,300	0.43
BuiltWithBot Disallow	2,838	0.28
Pipl Disallow	2,427	0.24
Youdao Bot Disallow	918	0.09
Google FeedFetcher Disallow	142	0.01
LLMS Text	133	0.01
Google Read Aloud Disallow	58	0.01
Google Web Light Disallow	15	0
Google BIG Bench Disallow	5	0