Robots.txt Usage Distribution in Saint Vincent and the Grenadines

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	2,090	20.48
Baidu Bot Disallow	1,766	17.31
Some Robots Disallow	1,506	14.76
Microsoft BingBot Disallow	315	3.09
Yandex Bot Disallow	308	3.02
Alexa Bot Disallow	307	3.01
Yahoo Slurp Disallow	302	2.96
Sogou Bot Disallow	298	2.92
Soso Bot Disallow	297	2.91
GPTBot Disallow	282	2.76
Robots Disallow	277	2.71
Common Crawl Bot Disallow	263	2.58
Google Bard Disallow	254	2.49
AppleBot Disallow	253	2.48
Anthropic Claude Bot Disallow	253	2.48
Facebook Bot Disallow	252	2.47
Google AdsBot Mobile Apps Disallow	252	2.47
Apple AI Bot Disallow	247	2.42
Cohere AI Disallow	240	2.35
Perplexity	238	2.33
Semrush Bot Disallow	49	0.48
Majestic Bot Disallow	43	0.42
Ahrefs Bot Disallow	34	0.33
Google Media Partners Disallow	32 ^*	0.31
Webz Omgili Disallow	15	0.15
ByteDance Bytespider Disallow	7	0.07
Exalead Bot Disallow	5	0.05
Microsoft MSNBot Disallow	3	0.03
DuckDuckGo Bot Disallow	3	0.03
360 Bot Disallow	2	0.02
Google News Bot Disallow	2	0.02
Diffbot Disallow	2	0.02
YOU Disallow	2	0.02
Google Image Bot Disallow	1	0.01
BuiltWithBot Disallow	1	0.01
Google Video Bot Disallow	1	0.01
Google Favicon Disallow	1	0.01
Google API Bot Disallow	1	0.01
Google DuplexWeb Disallow	1	0.01