Robots.txt Usage Distribution in Monaco

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	447	14.29
Google AdsBot Mobile Disallow	375	11.99
Googlebot Disallow	346	11.06
Robots Disallow	172	5.5
Ahrefs Bot Disallow	123	3.93
Google AdsBot Mobile Apps Disallow	111	3.55
GPTBot Disallow	102	3.26
Common Crawl Bot Disallow	102	3.26
Google Media Partners Disallow	101 ^*	3.23
Anthropic Claude Bot Disallow	94	3.01
Google Bard Disallow	93	2.97
Cohere AI Disallow	93	2.97
Perplexity	93	2.97
AppleBot Disallow	92	2.94
Facebook Bot Disallow	92	2.94
Apple AI Bot Disallow	92	2.94
Baidu Bot Disallow	68	2.17
Yandex Bot Disallow	55	1.76
Yahoo Slurp Disallow	47	1.5
Microsoft BingBot Disallow	46	1.47
Sogou Bot Disallow	41	1.31
Alexa Bot Disallow	37	1.18
Some Robots Disallow	36	1.15
Soso Bot Disallow	31	0.99
Google Image Bot Disallow	26	0.83
Majestic Bot Disallow	24	0.77
Microsoft MSNBot Disallow	23	0.74
Google News Bot Disallow	21	0.67
Google Video Bot Disallow	21	0.67
Google Favicon Disallow	21	0.67
DuckDuckGo Bot Disallow	19	0.61
Google API Bot Disallow	19	0.61
Google DuplexWeb Disallow	19	0.61
Semrush Bot Disallow	18	0.58
Exalead Bot Disallow	12	0.38
360 Bot Disallow	4	0.13
ByteDance Bytespider Disallow	3	0.1
BuiltWithBot Disallow	3	0.1
Youdao Bot Disallow	3	0.1
Webz Omgili Disallow	1	0.03
Diffbot Disallow	1	0.03
YOU Disallow	1	0.03