Robots.txt Usage Distribution in Hong Kong

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	19,455	18.41
Google AdsBot Mobile Disallow	13,655	12.92
Googlebot Disallow	13,623	12.89
Ahrefs Bot Disallow	6,923	6.55
Robots Disallow	4,493	4.25
Anthropic Claude Bot Disallow	3,469	14.01
GPTBot Disallow	3,155	12.75
Majestic Bot Disallow	2,992	2.83
AppleBot Disallow	2,837	2.69
Common Crawl Bot Disallow	2,821	11.4
Google Bard Disallow	2,618	10.58
Google AdsBot Mobile Apps Disallow	2,601	2.46
Facebook Bot Disallow	2,588	10.46
Apple AI Bot Disallow	2,515	10.16
Cohere AI Disallow	2,474	9.99
Perplexity	2,437	9.85
Baidu Bot Disallow	2,322	2.2
Yandex Bot Disallow	2,085	1.97
Some Robots Disallow	1,681	1.59
Microsoft BingBot Disallow	1,610	1.52
Alexa Bot Disallow	1,594	6.44
Sogou Bot Disallow	1,495	1.41
Semrush Bot Disallow	1,475	1.4
Yahoo Slurp Disallow	1,420	1.34
Soso Bot Disallow	1,174	1.11
Exalead Bot Disallow	299	0.28
Google Media Partners Disallow	291 ^*	0.28
Microsoft MSNBot Disallow	265	0.25
Google Image Bot Disallow	216	0.2
ByteDance Bytespider Disallow	207	0.84
Webz Omgili Disallow	194	0.78
DuckDuckGo Bot Disallow	191	0.18
360 Bot Disallow	85	0.08
Google News Bot Disallow	83	0.08
Google Video Bot Disallow	73	0.07
Youdao Bot Disallow	48	0.05
Google Favicon Disallow	41	0.04
Diffbot Disallow	37	0.15
BuiltWithBot Disallow	31	0.03
Google API Bot Disallow	29	0.03
Google DuplexWeb Disallow	28	0.03
YOU Disallow	15	0.06
Pipl Disallow	11	0.01
Google FeedFetcher Disallow	1	0