Robots.txt Usage Distribution in Antigua and Barbuda

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	379	12.49
Baidu Bot Disallow	306	10.09
Sogou Bot Disallow	287	9.46
Robots Disallow	186	6.13
Yandex Bot Disallow	173	5.7
Some Robots Disallow	171	5.64
Microsoft BingBot Disallow	164	5.41
Yahoo Slurp Disallow	164	5.41
Google AdsBot Disallow	158	5.21
Alexa Bot Disallow	154	27.16
Soso Bot Disallow	154	5.08
Google Media Partners Disallow	80 ^*	2.64
Ahrefs Bot Disallow	71	2.34
GPTBot Disallow	52	9.17
Anthropic Claude Bot Disallow	50	8.82
Google Bard Disallow	50	8.82
Common Crawl Bot Disallow	49	8.64
AppleBot Disallow	46	1.52
Facebook Bot Disallow	44	7.76
Apple AI Bot Disallow	42	7.41
Perplexity	41	7.23
Cohere AI Disallow	39	6.88
Google AdsBot Mobile Apps Disallow	39	1.29
Majestic Bot Disallow	25	0.82
Semrush Bot Disallow	25	0.82
ByteDance Bytespider Disallow	20	3.53
Microsoft MSNBot Disallow	9	0.3
Google Image Bot Disallow	9	0.3
Webz Omgili Disallow	8	1.41
360 Bot Disallow	5	0.16
Diffbot Disallow	4	0.71
YOU Disallow	4	0.71
Google News Bot Disallow	4	0.13
Google Video Bot Disallow	4	0.13
DuckDuckGo Bot Disallow	4	0.13
Google Favicon Disallow	4	0.13
Exalead Bot Disallow	3	0.1
Google API Bot Disallow	3	0.1
Google DuplexWeb Disallow	3	0.1
Google FeedFetcher Disallow	1	0.03