Robots.txt Usage Distribution in Mayotte

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	158	11.38
Robots Disallow	148	10.66
Baidu Bot Disallow	136	9.79
Sogou Bot Disallow	134	9.65
Some Robots Disallow	122	8.78
Yandex Bot Disallow	99	7.13
Yahoo Slurp Disallow	99	7.13
Microsoft BingBot Disallow	98	7.06
Alexa Bot Disallow	98	70
Soso Bot Disallow	97	6.98
Google AdsBot Disallow	40	2.88
Google Media Partners Disallow	40 ^*	2.88
Ahrefs Bot Disallow	26	1.87
Semrush Bot Disallow	13	0.94
Majestic Bot Disallow	9	0.65
GPTBot Disallow	8	5.71
Common Crawl Bot Disallow	6	4.29
Microsoft MSNBot Disallow	5	0.36
Anthropic Claude Bot Disallow	4	2.86
Google Bard Disallow	4	2.86
Google Image Bot Disallow	4	0.29
Facebook Bot Disallow	3	2.14
ByteDance Bytespider Disallow	3	2.14
Webz Omgili Disallow	3	2.14
AppleBot Disallow	2	0.14
Cohere AI Disallow	2	1.43
Apple AI Bot Disallow	2	1.43
Perplexity	2	1.43
Google AdsBot Mobile Apps Disallow	2	0.14
Exalead Bot Disallow	2	0.14
360 Bot Disallow	2	0.14
BuiltWithBot Disallow	2	0.14
Google News Bot Disallow	2	0.14
Google Video Bot Disallow	2	0.14
DuckDuckGo Bot Disallow	2	0.14
Google Favicon Disallow	2	0.14
Google API Bot Disallow	2	0.14
Google DuplexWeb Disallow	2	0.14
Diffbot Disallow	1	0.71
YOU Disallow	1	0.71
Youdao Bot Disallow	1	0.07
Pipl Disallow	1	0.07