Robots.txt Usage Distribution in Mali

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	86	9.38
Sogou Bot Disallow	84	9.16
Soso Bot Disallow	84	9.16
Alexa Bot Disallow	80	8.72
Yahoo Slurp Disallow	79	8.62
Microsoft BingBot Disallow	78	8.51
Robots Disallow	75	8.18
Googlebot Disallow	56	6.11
Google AdsBot Disallow	52	5.67
Google Media Partners Disallow	32 ^*	3.49
GPTBot Disallow	19	2.07
Baidu Bot Disallow	17	1.85
Ahrefs Bot Disallow	15	1.64
Common Crawl Bot Disallow	15	1.64
AppleBot Disallow	14	1.53
Facebook Bot Disallow	14	1.53
Anthropic Claude Bot Disallow	13	1.42
Google Bard Disallow	13	1.42
Apple AI Bot Disallow	13	1.42
Cohere AI Disallow	13	1.42
Perplexity	13	1.42
Google AdsBot Mobile Apps Disallow	12	1.31
Majestic Bot Disallow	12	1.31
Semrush Bot Disallow	11	1.2
ByteDance Bytespider Disallow	3	0.33
Microsoft MSNBot Disallow	2	0.22
Google Image Bot Disallow	2	0.22
360 Bot Disallow	2	0.22
Exalead Bot Disallow	1	0.11
Webz Omgili Disallow	1	0.11
Google News Bot Disallow	1	0.11
Google Video Bot Disallow	1	0.11
DuckDuckGo Bot Disallow	1	0.11
Diffbot Disallow	1	0.11
Youdao Bot Disallow	1	0.11
YOU Disallow	1	0.11