Robots.txt Usage Distribution in the United States Virgin Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	309	18.11
Anthropic Claude Bot Disallow	100	12.32
Common Crawl Bot Disallow	99	12.19
GPTBot Disallow	98	12.07
AppleBot Disallow	97	5.69
Facebook Bot Disallow	97	11.95
Google Bard Disallow	96	11.82
Apple AI Bot Disallow	93	11.45
Cohere AI Disallow	92	11.33
Perplexity	91	11.21
Google AdsBot Mobile Apps Disallow	89	5.22
Ahrefs Bot Disallow	70	4.1
Baidu Bot Disallow	50	2.93
Robots Disallow	49	2.87
Yandex Bot Disallow	35	2.05
Some Robots Disallow	33	1.93
Sogou Bot Disallow	30	1.76
Alexa Bot Disallow	25	3.08
Microsoft BingBot Disallow	23	1.35
Majestic Bot Disallow	23	1.35
Yahoo Slurp Disallow	22	1.29
Semrush Bot Disallow	22	1.29
Soso Bot Disallow	19	1.11
Google Media Partners Disallow	13 ^*	0.76
Microsoft MSNBot Disallow	6	0.35
ByteDance Bytespider Disallow	4	0.49
Exalead Bot Disallow	4	0.23
Webz Omgili Disallow	4	0.49
Google Image Bot Disallow	4	0.23
360 Bot Disallow	3	0.18
Diffbot Disallow	2	0.25
BuiltWithBot Disallow	2	0.12
YOU Disallow	1	0.12
Pipl Disallow	1	0.06