Robots.txt Nutzungsverteilung in der Dominikanischen Republik

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	1.239	11,46
Googlebot Disallow	1.051	9,73
Google AdsBot Mobile Disallow	695	6,43
Baidu Bot Disallow	558	5,16
Robots Disallow	558	5,16
AppleBot Disallow	492	4,55
Ahrefs Bot Disallow	484	4,48
Yandex Bot Disallow	461	4,27
Microsoft BingBot Disallow	443	4,1
Yahoo Slurp Disallow	439	4,06
GPTBot Disallow	389	13,72
Sogou Bot Disallow	269	2,49
Common Crawl Bot Disallow	268	9,45
Google Bard Disallow	264	9,31
Anthropic Claude Bot Disallow	261	9,21
Facebook Bot Disallow	258	9,1
Apple AI Bot Disallow	255	8,99
Perplexity	248	8,75
Cohere AI Disallow	247	8,71
Google AdsBot Mobile Apps Disallow	244	2,26
ByteDance Bytespider Disallow	241	8,5
Google Image Bot Disallow	238	2,2
Alexa Bot Disallow	212	7,48
Semrush Bot Disallow	208	1,92
Soso Bot Disallow	194	1,8
Majestic Bot Disallow	179	1,66
Some Robots Disallow	156	1,44
Google Media Partners Disallow	119 ^*	1,1
Microsoft MSNBot Disallow	36	0,33
Webz Omgili Disallow	25	0,88
Exalead Bot Disallow	11	0,1
Diffbot Disallow	8	0,28
DuckDuckGo Bot Disallow	8	0,07
YOU Disallow	7	0,25
Google News Bot Disallow	6	0,06
Google Video Bot Disallow	6	0,06
Google Favicon Disallow	6	0,06
Google API Bot Disallow	6	0,06
Google DuplexWeb Disallow	6	0,06
360 Bot Disallow	4	0,04
Pipl Disallow	4	0,04
Youdao Bot Disallow	3	0,03
Google FeedFetcher Disallow	1	0,01