Robots.txt Usage Distribution in Sudan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	81	31.89
Google Media Partners Disallow	13 ^*	5.12
Yandex Bot Disallow	11	4.33
Microsoft BingBot Disallow	10	3.94
Ahrefs Bot Disallow	10	3.94
Yahoo Slurp Disallow	9	3.54
GPTBot Disallow	9	12.33
Alexa Bot Disallow	8	10.96
Google Bard Disallow	8	10.96
Facebook Bot Disallow	8	10.96
Semrush Bot Disallow	8	3.15
Common Crawl Bot Disallow	7	9.59
AppleBot Disallow	7	2.76
Anthropic Claude Bot Disallow	7	9.59
Google AdsBot Mobile Apps Disallow	7	2.76
Soso Bot Disallow	6	2.36
Cohere AI Disallow	6	8.22
Apple AI Bot Disallow	6	8.22
Perplexity	5	6.85
Majestic Bot Disallow	5	1.97
Microsoft MSNBot Disallow	3	1.18
ByteDance Bytespider Disallow	2	2.74
Webz Omgili Disallow	2	2.74
Google Image Bot Disallow	2	0.79
Google News Bot Disallow	2	0.79
Google Video Bot Disallow	2	0.79
DuckDuckGo Bot Disallow	2	0.79
Google Favicon Disallow	2	0.79
Google API Bot Disallow	2	0.79
Google DuplexWeb Disallow	2	0.79
Diffbot Disallow	1	1.37
Youdao Bot Disallow	1	0.39