Robots.txt Usage Distribution in Taiwan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	14,534	10.98
Baidu Bot Disallow	11,060	8.35
Robots Disallow	10,109	7.64
Yandex Bot Disallow	9,493	7.17
Sogou Bot Disallow	8,147	6.15
Microsoft BingBot Disallow	7,806	5.9
Yahoo Slurp Disallow	7,787	5.88
Google AdsBot Disallow	7,769	5.87
Alexa Bot Disallow	7,617	36.25
Soso Bot Disallow	7,572	5.72
Majestic Bot Disallow	5,829	4.4
Ahrefs Bot Disallow	5,804	4.38
Google AdsBot Mobile Disallow	3,930	2.97
Semrush Bot Disallow	3,853	2.91
Anthropic Claude Bot Disallow	2,925	13.92
Some Robots Disallow	2,878	2.17
Common Crawl Bot Disallow	2,422	11.53
GPTBot Disallow	2,135	10.16
Google Image Bot Disallow	1,576	1.19
AppleBot Disallow	998	0.75
Google Bard Disallow	936	4.45
Facebook Bot Disallow	913	4.34
Google AdsBot Mobile Apps Disallow	869	0.66
Apple AI Bot Disallow	862	4.1
Cohere AI Disallow	791	3.76
Perplexity	791	3.76
ByteDance Bytespider Disallow	598	2.85
Microsoft MSNBot Disallow	447	0.34
Google Media Partners Disallow	388 ^*	0.29
360 Bot Disallow	365	0.28
Exalead Bot Disallow	261	0.2
Webz Omgili Disallow	240	1.14
Youdao Bot Disallow	115	0.09
Google Video Bot Disallow	104	0.08
Google News Bot Disallow	82	0.06
DuckDuckGo Bot Disallow	82	0.06
Google API Bot Disallow	75	0.06
Google Favicon Disallow	66	0.05
Google DuplexWeb Disallow	54	0.04
BuiltWithBot Disallow	39	0.03
YOU Disallow	29	0.14
Diffbot Disallow	15	0.07
Google FeedFetcher Disallow	8	0.01
Pipl Disallow	6	0
Google Read Aloud Disallow	1	0