Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	21.52
Googlebot Disallow	17,383,587	9.56
Baidu Bot Disallow	12,610,097	6.93
Google AdsBot Disallow	11,312,715	6.22
Google Media Partners Disallow	11,308,081	6.22
Google AdsBot Mobile Disallow	8,418,498	4.63
Robots Disallow	5,891,903	3.24
Yandex Bot Disallow	5,522,301	3.04
Microsoft BingBot Disallow	5,287,882	2.91
Ahrefs Bot Disallow	5,265,587	2.9
Alexa Bot Disallow	4,930,613	2.71
Yahoo Slurp Disallow	4,749,861	2.61
Sogou Bot Disallow	4,736,754	2.6
Soso Bot Disallow	4,511,509	2.48
GPTBot Disallow	3,762,344	2.07
Common Crawl Bot Disallow	3,668,558	2.02
AppleBot Disallow	3,342,388	1.84
Anthropic Claude Bot Disallow	3,255,079	1.79
Google Bard Disallow	3,131,120	1.72
Facebook Bot Disallow	3,091,616	1.7
Google AdsBot Mobile Apps Disallow	3,031,637	1.67
Apple AI Bot Disallow	2,931,218	1.61
Cohere AI Disallow	2,885,052	1.59
Perplexity	2,814,417	1.55
Majestic Bot Disallow	2,653,408	1.46
Semrush Bot Disallow	2,362,551	1.3
ByteDance Bytespider Disallow	698,026	0.38
Exalead Bot Disallow	540,249	0.3
Microsoft MSNBot Disallow	530,519	0.29
Google Image Bot Disallow	409,213	0.23
360 Bot Disallow	379,608	0.21
Webz Omgili Disallow	348,637	0.19
BuiltWithBot Disallow	157,562	0.09
Google News Bot Disallow	119,988	0.07
Google Video Bot Disallow	118,212	0.07
DuckDuckGo Bot Disallow	99,073	0.05
Diffbot Disallow	90,658	0.05
Google Favicon Disallow	90,125	0.05
Youdao Bot Disallow	84,260	0.05
Google API Bot Disallow	57,483	0.03
Google DuplexWeb Disallow	56,299	0.03
YOU Disallow	29,699	0.02
Pipl Disallow	20,941	0.01
Google FeedFetcher Disallow	8,123	0
Google Read Aloud Disallow	441	0
Google Web Light Disallow	109	0
Google BIG Bench Disallow	31	0