Robots.txt Usage Distribution in Indonesia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	24,219	12.11
Baidu Bot Disallow	18,684	9.34
Robots Disallow	16,545	8.27
Some Robots Disallow	15,696	7.85
Alexa Bot Disallow	13,497	40.65
Yahoo Slurp Disallow	12,439	6.22
Microsoft MSNBot Disallow	10,170	5.08
Google Image Bot Disallow	9,523	4.76
Yandex Bot Disallow	8,323	4.16
Ahrefs Bot Disallow	7,734	3.87
Majestic Bot Disallow	6,332	3.17
Google AdsBot Disallow	6,186	3.09
Sogou Bot Disallow	4,699	2.35
Semrush Bot Disallow	4,681	2.34
Google AdsBot Mobile Disallow	4,624	2.31
Microsoft BingBot Disallow	4,607	2.3
Soso Bot Disallow	4,567	2.28
Common Crawl Bot Disallow	3,763	11.33
GPTBot Disallow	3,586	10.8
Anthropic Claude Bot Disallow	2,989	9
Exalead Bot Disallow	1,687	0.84
Google Bard Disallow	1,511	4.55
ByteDance Bytespider Disallow	1,468	4.42
Youdao Bot Disallow	1,365	0.68
AppleBot Disallow	1,279	0.64
Google Media Partners Disallow	1,212 ^*	0.61
Facebook Bot Disallow	1,195	3.6
Google AdsBot Mobile Apps Disallow	1,149	0.57
Apple AI Bot Disallow	1,143	3.44
Perplexity	1,091	3.29
Webz Omgili Disallow	1,045	3.15
Cohere AI Disallow	1,029	3.1
Pipl Disallow	835	0.42
360 Bot Disallow	217	0.11
DuckDuckGo Bot Disallow	194	0.1
Google News Bot Disallow	137	0.07
BuiltWithBot Disallow	124	0.06
Google Video Bot Disallow	108	0.05
Google Favicon Disallow	93	0.05
Google API Bot Disallow	83	0.04
Google DuplexWeb Disallow	82	0.04
Diffbot Disallow	55	0.17
YOU Disallow	29	0.09
Google FeedFetcher Disallow	5	0
Google Web Light Disallow	1	0