Robots.txt Usage Distribution in East Timor

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	48	13.08
Robots Disallow	46	12.53
Some Robots Disallow	27	7.36
Yandex Bot Disallow	20	5.45
Microsoft BingBot Disallow	18	4.9
Alexa Bot Disallow	17	16.5
Yahoo Slurp Disallow	16	4.36
Sogou Bot Disallow	15	4.09
Soso Bot Disallow	15	4.09
Common Crawl Bot Disallow	13	12.62
Ahrefs Bot Disallow	12	3.27
GPTBot Disallow	10	9.71
Anthropic Claude Bot Disallow	10	9.71
Google Bard Disallow	10	9.71
Facebook Bot Disallow	10	9.71
AppleBot Disallow	9	2.45
Google AdsBot Mobile Apps Disallow	8	2.18
Apple AI Bot Disallow	8	7.77
Cohere AI Disallow	7	6.8
Perplexity	7	6.8
Majestic Bot Disallow	5	1.36
Semrush Bot Disallow	5	1.36
ByteDance Bytespider Disallow	4	3.88
Webz Omgili Disallow	4	3.88
Exalead Bot Disallow	3	0.82
Google Image Bot Disallow	3	0.82
Microsoft MSNBot Disallow	2	0.54
BuiltWithBot Disallow	2	0.54
Google News Bot Disallow	2	0.54
Google Video Bot Disallow	2	0.54
DuckDuckGo Bot Disallow	2	0.54
Google Favicon Disallow	2	0.54
Google API Bot Disallow	2	0.54
Google DuplexWeb Disallow	2	0.54
360 Bot Disallow	1	0.27