Robots.txt Nutzungsverteilung In Costa Rica

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	3.025	14,07
Google AdsBot Mobile Disallow	2.495	11,6
Googlebot Disallow	1.929	8,97
GPTBot Disallow	1.052	4,89
Common Crawl Bot Disallow	1.013	4,71
AppleBot Disallow	993	4,62
Anthropic Claude Bot Disallow	993	4,62
Google Bard Disallow	989	4,6
Facebook Bot Disallow	982	4,57
Apple AI Bot Disallow	976	4,54
Perplexity	971	4,52
Google AdsBot Mobile Apps Disallow	971	4,52
Cohere AI Disallow	967	4,5
Ahrefs Bot Disallow	651	3,03
Robots Disallow	612	2,85
Baidu Bot Disallow	346	1,61
Google Media Partners Disallow	338 ^*	1,57
Yandex Bot Disallow	292	1,36
Some Robots Disallow	279	1,3
Yahoo Slurp Disallow	232	1,08
Alexa Bot Disallow	214	1
Microsoft BingBot Disallow	209	0,97
Sogou Bot Disallow	207	0,96
Soso Bot Disallow	183	0,85
Semrush Bot Disallow	133	0,62
Majestic Bot Disallow	110	0,51
Microsoft MSNBot Disallow	71	0,33
Exalead Bot Disallow	58	0,27
ByteDance Bytespider Disallow	37	0,17
Google Image Bot Disallow	32	0,15
Webz Omgili Disallow	31	0,14
DuckDuckGo Bot Disallow	14	0,07
Google News Bot Disallow	12	0,06
Google Video Bot Disallow	12	0,06
Google Favicon Disallow	12	0,06
360 Bot Disallow	11	0,05
Google API Bot Disallow	11	0,05
Google DuplexWeb Disallow	11	0,05
Diffbot Disallow	8	0,04
BuiltWithBot Disallow	7	0,03
YOU Disallow	7	0,03
Youdao Bot Disallow	7	0,03
Pipl Disallow	3	0,01