Robots.txt Usage Distribution in Kenya

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	1,698	10.88
Robots Disallow	1,644	10.53
Google AdsBot Disallow	1,218	7.8
Baidu Bot Disallow	944	6.05
Google AdsBot Mobile Disallow	930	5.96
Sogou Bot Disallow	868	5.56
Yandex Bot Disallow	672	4.3
Microsoft BingBot Disallow	662	4.24
Yahoo Slurp Disallow	611	3.91
Alexa Bot Disallow	609	18.55
Soso Bot Disallow	584	3.74
Google Media Partners Disallow	584 ^*	3.74
Some Robots Disallow	531	3.4
Ahrefs Bot Disallow	395	2.53
GPTBot Disallow	388	11.82
Common Crawl Bot Disallow	335	10.2
Google Bard Disallow	317	9.66
Anthropic Claude Bot Disallow	316	9.63
AppleBot Disallow	310	1.99
Facebook Bot Disallow	304	9.26
Apple AI Bot Disallow	300	9.14
Perplexity	298	9.08
Cohere AI Disallow	297	9.05
Google AdsBot Mobile Apps Disallow	292	1.87
Semrush Bot Disallow	113	0.72
Majestic Bot Disallow	110	0.7
Microsoft MSNBot Disallow	38	0.24
Google Image Bot Disallow	36	0.23
ByteDance Bytespider Disallow	35	1.07
Webz Omgili Disallow	25	0.76
Exalead Bot Disallow	20	0.13
Google News Bot Disallow	17	0.11
Diffbot Disallow	15	0.46
Google Video Bot Disallow	14	0.09
Google Favicon Disallow	14	0.09
YOU Disallow	12	0.37
DuckDuckGo Bot Disallow	12	0.08
Google API Bot Disallow	12	0.08
Google DuplexWeb Disallow	12	0.08
360 Bot Disallow	9	0.06
Youdao Bot Disallow	4	0.03
BuiltWithBot Disallow	3	0.02
Pipl Disallow	1	0.01
Google FeedFetcher Disallow	1	0.01