Robots.txt Usage Distribution in Mayotte

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	218	13.74
Baidu Bot Disallow	181	11.41
Sogou Bot Disallow	176	11.09
Robots Disallow	158	9.96
Yandex Bot Disallow	112	7.06
Alexa Bot Disallow	112	7.06
Yahoo Slurp Disallow	112	7.06
Microsoft BingBot Disallow	111	6.99
Soso Bot Disallow	111	6.99
Some Robots Disallow	100	6.3
Google AdsBot Disallow	52	3.28
Google Media Partners Disallow	29 ^*	1.83
Ahrefs Bot Disallow	25	1.58
Semrush Bot Disallow	15	0.95
Majestic Bot Disallow	11	0.69
GPTBot Disallow	6	0.38
Common Crawl Bot Disallow	5	0.32
Microsoft MSNBot Disallow	5	0.32
Google Image Bot Disallow	4	0.25
Anthropic Claude Bot Disallow	3	0.19
Exalead Bot Disallow	3	0.19
Webz Omgili Disallow	3	0.19
360 Bot Disallow	3	0.19
Google Bard Disallow	2	0.13
Facebook Bot Disallow	2	0.13
Cohere AI Disallow	2	0.13
Google AdsBot Mobile Apps Disallow	2	0.13
ByteDance Bytespider Disallow	2	0.13
BuiltWithBot Disallow	2	0.13
Google News Bot Disallow	2	0.13
Google Video Bot Disallow	2	0.13
DuckDuckGo Bot Disallow	2	0.13
Google Favicon Disallow	2	0.13
Google API Bot Disallow	2	0.13
Google DuplexWeb Disallow	2	0.13
Pipl Disallow	2	0.13
AppleBot Disallow	1	0.06
Apple AI Bot Disallow	1	0.06
Perplexity	1	0.06
Diffbot Disallow	1	0.06
YOU Disallow	1	0.06
Youdao Bot Disallow	1	0.06