Robots.txt Nutzungsverteilung in Tansania

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	593	12,33
Robots Disallow	529	11
Google AdsBot Disallow	497	10,33
Google AdsBot Mobile Disallow	457	9,5
Baidu Bot Disallow	221	4,59
Google Media Partners Disallow	159 ^*	3,31
Yandex Bot Disallow	159	3,31
Microsoft BingBot Disallow	143	2,97
GPTBot Disallow	140	12,25
Some Robots Disallow	135	2,81
Common Crawl Bot Disallow	131	11,46
Google Bard Disallow	129	11,29
Yahoo Slurp Disallow	127	2,64
Google AdsBot Mobile Apps Disallow	124	2,58
Anthropic Claude Bot Disallow	122	10,67
Alexa Bot Disallow	121	10,59
AppleBot Disallow	120	2,49
Facebook Bot Disallow	120	10,5
Perplexity	119	10,41
Cohere AI Disallow	118	10,32
Apple AI Bot Disallow	117	10,24
Soso Bot Disallow	115	2,39
Ahrefs Bot Disallow	86	1,79
Semrush Bot Disallow	49	1,02
Majestic Bot Disallow	46	0,96
Microsoft MSNBot Disallow	21	0,44
Google Image Bot Disallow	16	0,33
Exalead Bot Disallow	12	0,25
Google News Bot Disallow	11	0,23
Google Video Bot Disallow	11	0,23
Google Favicon Disallow	11	0,23
ByteDance Bytespider Disallow	8	0,7
DuckDuckGo Bot Disallow	8	0,17
Google API Bot Disallow	8	0,17
Google DuplexWeb Disallow	8	0,17
360 Bot Disallow	6	0,12
Webz Omgili Disallow	5	0,44
BuiltWithBot Disallow	3	0,06
YOU Disallow	2	0,17
Youdao Bot Disallow	2	0,04
Diffbot Disallow	1	0,09