Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	21.52
Googlebot Disallow	17,383,587	9.56
Baidu Bot Disallow	12,610,097	6.93
Google AdsBot Disallow	11,312,715	6.22
Google Media Partners Disallow	11,308,081	6.22
Google AdsBot Mobile Disallow	8,418,498	4.63
Robots Disallow	5,891,903	3.24
Yandex Bot Disallow	5,522,301	3.04
Microsoft BingBot Disallow	5,287,882	2.91
Ahrefs Bot Disallow	5,265,587	2.9
Alexa Bot Disallow	4,930,613	15.19
Yahoo Slurp Disallow	4,749,861	2.61
Sogou Bot Disallow	4,736,754	2.6
Soso Bot Disallow	4,511,509	2.48
GPTBot Disallow	3,762,344	11.59
Common Crawl Bot Disallow	3,668,558	11.3
AppleBot Disallow	3,342,388	1.84
Anthropic Claude Bot Disallow	3,255,079	10.03
Google Bard Disallow	3,131,120	9.65
Facebook Bot Disallow	3,091,616	9.53
Google AdsBot Mobile Apps Disallow	3,031,637	1.67
Apple AI Bot Disallow	2,931,218	9.03
Cohere AI Disallow	2,885,052	8.89
Perplexity	2,814,417	8.67
Majestic Bot Disallow	2,653,408	1.46
Semrush Bot Disallow	2,362,551	1.3
ByteDance Bytespider Disallow	698,026	2.15
Exalead Bot Disallow	540,249	0.3
Microsoft MSNBot Disallow	530,519	0.29
Google Image Bot Disallow	409,213	0.23
360 Bot Disallow	379,608	0.21
Webz Omgili Disallow	348,637	1.07
BuiltWithBot Disallow	157,562	0.09
Google News Bot Disallow	119,988	0.07
Google Video Bot Disallow	118,212	0.07
DuckDuckGo Bot Disallow	99,073	0.05
Diffbot Disallow	90,658	0.28
Google Favicon Disallow	90,125	0.05
Youdao Bot Disallow	84,260	0.05
Google API Bot Disallow	57,483	0.03
Google DuplexWeb Disallow	56,299	0.03
YOU Disallow	29,699	0.09
Pipl Disallow	20,941	0.01
Google FeedFetcher Disallow	8,123	0
Google Read Aloud Disallow	441	0
Google Web Light Disallow	109	0
Google BIG Bench Disallow	31	0