Robots.txt Usage Distribution in Yemen

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	43	14.98
Yandex Bot Disallow	27	9.41
Alexa Bot Disallow	23	36.51
Microsoft BingBot Disallow	22	7.67
Yahoo Slurp Disallow	21	7.32
Ahrefs Bot Disallow	19	6.62
Sogou Bot Disallow	19	6.62
Soso Bot Disallow	19	6.62
Majestic Bot Disallow	14	4.88
Semrush Bot Disallow	10	3.48
Common Crawl Bot Disallow	7	11.11
GPTBot Disallow	6	9.52
AppleBot Disallow	5	1.74
Google Bard Disallow	5	7.94
Google AdsBot Mobile Apps Disallow	5	1.74
Anthropic Claude Bot Disallow	4	6.35
Facebook Bot Disallow	4	6.35
Apple AI Bot Disallow	4	6.35
Cohere AI Disallow	4	6.35
Perplexity	4	6.35
Microsoft MSNBot Disallow	4	1.39
Google Image Bot Disallow	4	1.39
Exalead Bot Disallow	3	1.05
ByteDance Bytespider Disallow	2	3.17
Google Video Bot Disallow	2	0.7
360 Bot Disallow	1	0.35
BuiltWithBot Disallow	1	0.35
Google News Bot Disallow	1	0.35
DuckDuckGo Bot Disallow	1	0.35
Google Favicon Disallow	1	0.35
Google API Bot Disallow	1	0.35
Google DuplexWeb Disallow	1	0.35