Robots.txt Usage Distribution in Turks and Caicos Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	104	9.52
Google AdsBot Disallow	89	8.14
Robots Disallow	73	6.68
Some Robots Disallow	61	5.58
Baidu Bot Disallow	61	5.58
Google Media Partners Disallow	59 ^*	5.4
GPTBot Disallow	48	4.39
Sogou Bot Disallow	47	4.3
Common Crawl Bot Disallow	45	4.12
Google Bard Disallow	44	4.03
Anthropic Claude Bot Disallow	43	3.93
Facebook Bot Disallow	42	3.84
AppleBot Disallow	41	3.75
Apple AI Bot Disallow	41	3.75
Google AdsBot Mobile Apps Disallow	41	3.75
Perplexity	40	3.66
Cohere AI Disallow	39	3.57
Yandex Bot Disallow	30	2.74
Alexa Bot Disallow	24	2.2
Yahoo Slurp Disallow	24	2.2
Microsoft BingBot Disallow	22	2.01
Soso Bot Disallow	20	1.83
Ahrefs Bot Disallow	12	1.1
Microsoft MSNBot Disallow	7	0.64
Majestic Bot Disallow	5	0.46
Google Image Bot Disallow	4	0.37
Semrush Bot Disallow	3	0.27
ByteDance Bytespider Disallow	3	0.27
Webz Omgili Disallow	3	0.27
Exalead Bot Disallow	2	0.18
360 Bot Disallow	2	0.18
Google News Bot Disallow	2	0.18
Google Video Bot Disallow	2	0.18
DuckDuckGo Bot Disallow	2	0.18
Google Favicon Disallow	2	0.18
Google API Bot Disallow	2	0.18
Google DuplexWeb Disallow	2	0.18
BuiltWithBot Disallow	1	0.09
Google FeedFetcher Disallow	1	0.09