Robots.txt Nutzungsverteilung in Antigua und Barbuda

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	415	12,91
Baidu Bot Disallow	323	10,05
Sogou Bot Disallow	262	8,15
Some Robots Disallow	190	5,91
Robots Disallow	190	5,91
Yandex Bot Disallow	188	5,85
Google AdsBot Disallow	185	5,76
Microsoft BingBot Disallow	181	5,63
Yahoo Slurp Disallow	178	5,54
Alexa Bot Disallow	167	5,2
Soso Bot Disallow	166	5,16
Google Media Partners Disallow	88 ^*	2,74
Ahrefs Bot Disallow	75	2,33
Anthropic Claude Bot Disallow	50	1,56
GPTBot Disallow	48	1,49
Common Crawl Bot Disallow	47	1,46
Google Bard Disallow	47	1,46
AppleBot Disallow	46	1,43
Facebook Bot Disallow	45	1,4
Apple AI Bot Disallow	43	1,34
Perplexity	42	1,31
Google AdsBot Mobile Apps Disallow	42	1,31
Cohere AI Disallow	40	1,24
Majestic Bot Disallow	26	0,81
Semrush Bot Disallow	23	0,72
ByteDance Bytespider Disallow	21	0,65
Google Image Bot Disallow	13	0,4
Microsoft MSNBot Disallow	12	0,37
Webz Omgili Disallow	8	0,25
Google News Bot Disallow	6	0,19
Google Video Bot Disallow	6	0,19
DuckDuckGo Bot Disallow	6	0,19
Google Favicon Disallow	6	0,19
Exalead Bot Disallow	5	0,16
360 Bot Disallow	5	0,16
Google API Bot Disallow	5	0,16
Google DuplexWeb Disallow	5	0,16
Diffbot Disallow	4	0,12
YOU Disallow	4	0,12
Google FeedFetcher Disallow	1	0,03