Robots.txt Nutzungsverteilung in Afghanistan

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	172	8,89
Robots Disallow	160	8,27
Baidu Bot Disallow	134	6,93
Sogou Bot Disallow	128	6,62
Yandex Bot Disallow	124	6,41
Ahrefs Bot Disallow	113	5,84
Microsoft BingBot Disallow	105	5,43
Google AdsBot Disallow	100	5,17
Majestic Bot Disallow	87	4,5
Alexa Bot Disallow	86	28,1
Google AdsBot Mobile Disallow	74	3,83
Soso Bot Disallow	72	3,72
Yahoo Slurp Disallow	70	3,62
Some Robots Disallow	65	3,36
Semrush Bot Disallow	44	2,28
Google Media Partners Disallow	40 ^*	2,07
Common Crawl Bot Disallow	37	12,09
AppleBot Disallow	31	1,6
Exalead Bot Disallow	31	1,6
GPTBot Disallow	26	8,5
Anthropic Claude Bot Disallow	23	7,52
Google Bard Disallow	23	7,52
Facebook Bot Disallow	22	7,19
Apple AI Bot Disallow	22	7,19
Perplexity	20	6,54
Cohere AI Disallow	19	6,21
Google AdsBot Mobile Apps Disallow	19	0,98
360 Bot Disallow	18	0,93
Microsoft MSNBot Disallow	15	0,78
Google Image Bot Disallow	11	0,57
BuiltWithBot Disallow	11	0,57
ByteDance Bytespider Disallow	7	2,29
Webz Omgili Disallow	5	1,63
DuckDuckGo Bot Disallow	4	0,21
Google News Bot Disallow	3	0,16
Google Video Bot Disallow	3	0,16
Diffbot Disallow	2	0,65
YOU Disallow	2	0,65
Google Favicon Disallow	2	0,1
Youdao Bot Disallow	2	0,1
Google API Bot Disallow	1	0,05
Google DuplexWeb Disallow	1	0,05