Robots.txt Usage Distribution in Madagascar

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	155	14.62
Googlebot Disallow	137	12.92
Google AdsBot Disallow	107	10.09
Google AdsBot Mobile Disallow	97	9.15
Baidu Bot Disallow	34	3.21
Common Crawl Bot Disallow	34	3.21
GPTBot Disallow	31	2.92
Some Robots Disallow	30	2.83
Google Bard Disallow	28	2.64
AppleBot Disallow	27	2.55
Anthropic Claude Bot Disallow	27	2.55
Facebook Bot Disallow	27	2.55
Apple AI Bot Disallow	27	2.55
Perplexity	27	2.55
Microsoft BingBot Disallow	26	2.45
Cohere AI Disallow	26	2.45
Yahoo Slurp Disallow	24	2.26
Google AdsBot Mobile Apps Disallow	24	2.26
Yandex Bot Disallow	23	2.17
Alexa Bot Disallow	20	1.89
Ahrefs Bot Disallow	20	1.89
Soso Bot Disallow	18	1.7
Google Media Partners Disallow	16 ^*	1.51
Majestic Bot Disallow	15	1.42
Semrush Bot Disallow	14	1.32
ByteDance Bytespider Disallow	6	0.57
DuckDuckGo Bot Disallow	6	0.57
Exalead Bot Disallow	5	0.47
Webz Omgili Disallow	4	0.38
Google Image Bot Disallow	4	0.38
Microsoft MSNBot Disallow	3	0.28
360 Bot Disallow	3	0.28
Diffbot Disallow	3	0.28
BuiltWithBot Disallow	3	0.28
YOU Disallow	3	0.28
Google News Bot Disallow	2	0.19
Google Video Bot Disallow	1	0.09
Google Favicon Disallow	1	0.09
Google API Bot Disallow	1	0.09
Google DuplexWeb Disallow	1	0.09