Robots.txt Usage Distribution in Mayotte

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	202	12.14
Baidu Bot Disallow	165	9.92
Robots Disallow	157	9.44
Sogou Bot Disallow	156	9.38
Yandex Bot Disallow	145	8.71
Alexa Bot Disallow	144	77.42
Yahoo Slurp Disallow	144	8.65
Microsoft BingBot Disallow	143	8.59
Soso Bot Disallow	143	8.59
Some Robots Disallow	56	3.37
Google AdsBot Disallow	47	2.82
Google Media Partners Disallow	34 ^*	2.04
Ahrefs Bot Disallow	29	1.74
Semrush Bot Disallow	15	0.9
Majestic Bot Disallow	10	0.6
GPTBot Disallow	7	3.76
Common Crawl Bot Disallow	6	3.23
Microsoft MSNBot Disallow	5	0.3
Anthropic Claude Bot Disallow	4	2.15
Webz Omgili Disallow	4	2.15
Google Image Bot Disallow	4	0.24
Google Bard Disallow	3	1.61
Facebook Bot Disallow	3	1.61
ByteDance Bytespider Disallow	3	1.61
Exalead Bot Disallow	3	0.18
360 Bot Disallow	3	0.18
AppleBot Disallow	2	0.12
Apple AI Bot Disallow	2	1.08
Cohere AI Disallow	2	1.08
Perplexity	2	1.08
Google AdsBot Mobile Apps Disallow	2	0.12
BuiltWithBot Disallow	2	0.12
Google News Bot Disallow	2	0.12
Google Video Bot Disallow	2	0.12
DuckDuckGo Bot Disallow	2	0.12
Google Favicon Disallow	2	0.12
Google API Bot Disallow	2	0.12
Google DuplexWeb Disallow	2	0.12
Pipl Disallow	2	0.12
Diffbot Disallow	1	0.54
YOU Disallow	1	0.54
Youdao Bot Disallow	1	0.06