Robots.txt Usage Distribution in Monaco

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	368	14.4
Google AdsBot Mobile Disallow	307	12.02
Googlebot Disallow	285	11.15
Robots Disallow	179	7.01
Ahrefs Bot Disallow	108	4.23
Google AdsBot Mobile Apps Disallow	96	3.76
GPTBot Disallow	89	3.48
Common Crawl Bot Disallow	87	3.41
Anthropic Claude Bot Disallow	79	3.09
AppleBot Disallow	78	3.05
Google Bard Disallow	78	3.05
Cohere AI Disallow	78	3.05
Perplexity	78	3.05
Facebook Bot Disallow	77	3.01
Apple AI Bot Disallow	77	3.01
Baidu Bot Disallow	57	2.23
Yandex Bot Disallow	46	1.8
Google Media Partners Disallow	40 ^*	1.57
Microsoft BingBot Disallow	39	1.53
Yahoo Slurp Disallow	39	1.53
Alexa Bot Disallow	30	1.17
Soso Bot Disallow	27	1.06
Google Image Bot Disallow	21	0.82
Majestic Bot Disallow	20	0.78
Microsoft MSNBot Disallow	19	0.74
Google News Bot Disallow	18	0.7
Google Video Bot Disallow	18	0.7
DuckDuckGo Bot Disallow	18	0.7
Google Favicon Disallow	18	0.7
Google API Bot Disallow	18	0.7
Google DuplexWeb Disallow	18	0.7
Semrush Bot Disallow	17	0.67
Exalead Bot Disallow	11	0.43
360 Bot Disallow	4	0.16
ByteDance Bytespider Disallow	3	0.12
BuiltWithBot Disallow	3	0.12
Youdao Bot Disallow	3	0.12
YOU Disallow	2	0.08
Webz Omgili Disallow	1	0.04
Diffbot Disallow	1	0.04