Robots.txt Usage Distribution in Trinidad and Tobago

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	254	10.46
Google AdsBot Mobile Disallow	225	9.26
Googlebot Disallow	156	6.42
GPTBot Disallow	139	5.72
Common Crawl Bot Disallow	134	5.52
Google AdsBot Mobile Apps Disallow	134	5.52
Google Bard Disallow	129	5.31
Anthropic Claude Bot Disallow	128	5.27
Facebook Bot Disallow	128	5.27
AppleBot Disallow	127	5.23
Apple AI Bot Disallow	126	5.19
Cohere AI Disallow	126	5.19
Perplexity	124	5.1
Robots Disallow	112	4.61
Baidu Bot Disallow	46	1.89
Ahrefs Bot Disallow	41	1.69
Yahoo Slurp Disallow	37	1.52
Yandex Bot Disallow	35	1.44
Microsoft BingBot Disallow	30	1.24
Alexa Bot Disallow	28	1.15
Some Robots Disallow	26	1.07
Sogou Bot Disallow	22	0.91
Soso Bot Disallow	21	0.86
Microsoft MSNBot Disallow	18	0.74
Google Image Bot Disallow	17	0.7
Majestic Bot Disallow	14	0.58
Semrush Bot Disallow	13	0.54
Google News Bot Disallow	7	0.29
Google Video Bot Disallow	7	0.29
Webz Omgili Disallow	6	0.25
Exalead Bot Disallow	4	0.16
Google Favicon Disallow	4	0.16
ByteDance Bytespider Disallow	2	0.08
360 Bot Disallow	2	0.08
DuckDuckGo Bot Disallow	2	0.08
YOU Disallow	2	0.08
Diffbot Disallow	1	0.04
Google API Bot Disallow	1	0.04
Google DuplexWeb Disallow	1	0.04