Robots.txt Usage Distribution in Christmas Island

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	694	16.81
Baidu Bot Disallow	657	15.92
Some Robots Disallow	545	13.2
Robots Disallow	498	12.06
Yandex Bot Disallow	175	4.24
Microsoft BingBot Disallow	175	4.24
Yahoo Slurp Disallow	163	3.95
Alexa Bot Disallow	158	34.5
Sogou Bot Disallow	150	3.63
Soso Bot Disallow	148	3.59
Google AdsBot Disallow	66	1.6
Google Media Partners Disallow	65 ^*	1.57
Ahrefs Bot Disallow	58	1.41
GPTBot Disallow	55	12.01
Google AdsBot Mobile Disallow	43	1.04
Semrush Bot Disallow	42	1.02
Majestic Bot Disallow	41	0.99
Common Crawl Bot Disallow	37	8.08
Google Bard Disallow	31	6.77
Anthropic Claude Bot Disallow	30	6.55
Facebook Bot Disallow	29	6.33
AppleBot Disallow	22	0.53
Google AdsBot Mobile Apps Disallow	22	0.53
Cohere AI Disallow	21	4.59
Microsoft MSNBot Disallow	21	0.51
Apple AI Bot Disallow	20	4.37
Perplexity	19	4.15
Webz Omgili Disallow	18	3.93
Exalead Bot Disallow	16	0.39
ByteDance Bytespider Disallow	15	3.28
Google Image Bot Disallow	13	0.31
DuckDuckGo Bot Disallow	11	0.27
Diffbot Disallow	10	2.18
Google News Bot Disallow	9	0.22
Google Video Bot Disallow	9	0.22
Google Favicon Disallow	9	0.22
Google API Bot Disallow	8	0.19
Google DuplexWeb Disallow	8	0.19
360 Bot Disallow	6	0.15
YOU Disallow	4	0.87
BuiltWithBot Disallow	3	0.07
Pipl Disallow	3	0.07
Youdao Bot Disallow	1	0.02