Robots.txt Usage Distribution in Turks and Caicos Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	100	7.92
Googlebot Disallow	98	7.76
Robots Disallow	74	5.86
GPTBot Disallow	73	5.78
Common Crawl Bot Disallow	71	5.62
Google AdsBot Mobile Apps Disallow	67	5.3
Google Bard Disallow	61	4.83
Anthropic Claude Bot Disallow	60	4.75
Facebook Bot Disallow	60	4.75
AppleBot Disallow	59	4.67
Apple AI Bot Disallow	59	4.67
Perplexity	58	4.59
Baidu Bot Disallow	57	4.51
Cohere AI Disallow	57	4.51
Some Robots Disallow	51	4.04
Google Media Partners Disallow	48 ^*	3.8
Yandex Bot Disallow	30	2.38
Alexa Bot Disallow	25	1.98
Yahoo Slurp Disallow	25	1.98
Microsoft BingBot Disallow	22	1.74
Sogou Bot Disallow	21	1.66
Soso Bot Disallow	21	1.66
Ahrefs Bot Disallow	15	1.19
Majestic Bot Disallow	8	0.63
Microsoft MSNBot Disallow	8	0.63
Semrush Bot Disallow	6	0.48
Exalead Bot Disallow	4	0.32
ByteDance Bytespider Disallow	3	0.24
360 Bot Disallow	3	0.24
Webz Omgili Disallow	3	0.24
Google Image Bot Disallow	2	0.16
BuiltWithBot Disallow	2	0.16
Google News Bot Disallow	2	0.16
Google Video Bot Disallow	2	0.16
DuckDuckGo Bot Disallow	2	0.16
Google Favicon Disallow	2	0.16
Google API Bot Disallow	2	0.16
Google DuplexWeb Disallow	2	0.16