Robots.txt Usage Distribution in Costa Rica

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	2,258	14.02
Google AdsBot Mobile Disallow	1,862	11.56
Googlebot Disallow	1,525	9.47
GPTBot Disallow	731	12.52
Common Crawl Bot Disallow	704	12.06
AppleBot Disallow	692	4.3
Anthropic Claude Bot Disallow	691	11.84
Google Bard Disallow	688	11.78
Facebook Bot Disallow	682	11.68
Apple AI Bot Disallow	681	11.66
Perplexity	678	11.61
Cohere AI Disallow	676	11.58
Google AdsBot Mobile Apps Disallow	676	4.2
Robots Disallow	570	3.54
Ahrefs Bot Disallow	490	3.04
Baidu Bot Disallow	314	1.95
Google Media Partners Disallow	279 ^*	1.73
Yandex Bot Disallow	244	1.52
Sogou Bot Disallow	236	1.47
Some Robots Disallow	233	1.45
Yahoo Slurp Disallow	204	1.27
Alexa Bot Disallow	193	3.31
Microsoft BingBot Disallow	187	1.16
Soso Bot Disallow	164	1.02
Semrush Bot Disallow	92	0.57
Majestic Bot Disallow	88	0.55
Microsoft MSNBot Disallow	52	0.32
Exalead Bot Disallow	41	0.25
ByteDance Bytespider Disallow	31	0.53
Google Image Bot Disallow	27	0.17
Webz Omgili Disallow	20	0.34
Google News Bot Disallow	11	0.07
Google Video Bot Disallow	11	0.07
DuckDuckGo Bot Disallow	11	0.07
Diffbot Disallow	10	0.17
Google Favicon Disallow	10	0.06
Google API Bot Disallow	9	0.06
Google DuplexWeb Disallow	9	0.06
YOU Disallow	8	0.14
360 Bot Disallow	6	0.04
Youdao Bot Disallow	5	0.03
BuiltWithBot Disallow	3	0.02
Pipl Disallow	1	0.01