Robots.txt Usage Distribution in Monaco

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	470	15.08
Google AdsBot Mobile Disallow	392	12.58
Googlebot Disallow	363	11.65
Robots Disallow	153	4.91
Ahrefs Bot Disallow	118	3.79
Google AdsBot Mobile Apps Disallow	112	3.59
Common Crawl Bot Disallow	103	3.31
GPTBot Disallow	102	3.27
Google Media Partners Disallow	97 ^*	3.11
Anthropic Claude Bot Disallow	91	2.92
Google Bard Disallow	91	2.92
Facebook Bot Disallow	90	2.89
Cohere AI Disallow	90	2.89
AppleBot Disallow	88	2.82
Apple AI Bot Disallow	87	2.79
Perplexity	87	2.79
Baidu Bot Disallow	68	2.18
Yandex Bot Disallow	54	1.73
Yahoo Slurp Disallow	48	1.54
Microsoft BingBot Disallow	47	1.51
Alexa Bot Disallow	40	1.28
Some Robots Disallow	39	1.25
Soso Bot Disallow	33	1.06
Sogou Bot Disallow	31	0.99
Google Image Bot Disallow	25	0.8
Majestic Bot Disallow	23	0.74
Microsoft MSNBot Disallow	22	0.71
Google News Bot Disallow	20	0.64
Google Video Bot Disallow	20	0.64
Google Favicon Disallow	20	0.64
DuckDuckGo Bot Disallow	18	0.58
Google API Bot Disallow	18	0.58
Google DuplexWeb Disallow	18	0.58
Semrush Bot Disallow	15	0.48
Exalead Bot Disallow	13	0.42
Youdao Bot Disallow	4	0.13
360 Bot Disallow	2	0.06
BuiltWithBot Disallow	2	0.06
ByteDance Bytespider Disallow	1	0.03
Webz Omgili Disallow	1	0.03