Robots.txt Nutzungsverteilung in Mexiko

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	55.427	19,14
Googlebot Disallow	41.903	14,47
Google AdsBot Mobile Disallow	35.483	12,25
Ahrefs Bot Disallow	16.918	5,84
Baidu Bot Disallow	11.409	3,94
Robots Disallow	9.451	3,26
Sogou Bot Disallow	8.873	3,06
Anthropic Claude Bot Disallow	8.758	3,02
Yandex Bot Disallow	7.243	2,5
Microsoft BingBot Disallow	7.019	2,42
Yahoo Slurp Disallow	6.911	2,39
Alexa Bot Disallow	6.897	2,38
GPTBot Disallow	6.848	2,36
Soso Bot Disallow	6.410	2,21
Some Robots Disallow	6.014	2,08
Common Crawl Bot Disallow	5.886	2,03
AppleBot Disallow	5.734	1,98
Google Bard Disallow	5.682	1,96
Facebook Bot Disallow	5.626	1,94
Apple AI Bot Disallow	5.574	1,92
Perplexity	5.495	1,9
Cohere AI Disallow	5.433	1,88
Google AdsBot Mobile Apps Disallow	5.237	1,81
Semrush Bot Disallow	2.277	0,79
Google Media Partners Disallow	1.449 ^*	0,5
Majestic Bot Disallow	1.413	0,49
Google Image Bot Disallow	863	0,3
Microsoft MSNBot Disallow	747	0,26
Webz Omgili Disallow	507	0,18
ByteDance Bytespider Disallow	453	0,16
Exalead Bot Disallow	311	0,11
Diffbot Disallow	272	0,09
YOU Disallow	245	0,08
360 Bot Disallow	136	0,05
Google News Bot Disallow	105	0,04
DuckDuckGo Bot Disallow	88	0,03
Google DuplexWeb Disallow	85	0,03
Youdao Bot Disallow	84	0,03
Google Video Bot Disallow	79	0,03
Google API Bot Disallow	66	0,02
Google Favicon Disallow	61	0,02
BuiltWithBot Disallow	46	0,02
Pipl Disallow	38	0,01
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0