Robots.txt Nutzungsverteilung In St. Helena

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	1.833	14,58
Baidu Bot Disallow	1.735	13,8
Some Robots Disallow	1.480	11,78
Sogou Bot Disallow	1.238	9,85
Robots Disallow	1.113	8,86
Yandex Bot Disallow	590	4,69
Microsoft BingBot Disallow	575	4,57
Yahoo Slurp Disallow	559	4,45
Alexa Bot Disallow	558	4,44
Soso Bot Disallow	537	4,27
Google Media Partners Disallow	338 ^*	2,69
Ahrefs Bot Disallow	191	1,52
GPTBot Disallow	177	1,41
Google AdsBot Disallow	171	1,36
Majestic Bot Disallow	123	0,98
Common Crawl Bot Disallow	116	0,92
Semrush Bot Disallow	116	0,92
Google Bard Disallow	114	0,91
Anthropic Claude Bot Disallow	99	0,79
AppleBot Disallow	95	0,76
Facebook Bot Disallow	94	0,75
Google AdsBot Mobile Disallow	84	0,67
Apple AI Bot Disallow	78	0,62
Webz Omgili Disallow	74	0,59
Perplexity	67	0,53
Cohere AI Disallow	65	0,52
ByteDance Bytespider Disallow	60	0,48
Diffbot Disallow	44	0,35
YOU Disallow	35	0,28
Google AdsBot Mobile Apps Disallow	33	0,26
Microsoft MSNBot Disallow	29	0,23
Google Image Bot Disallow	25	0,2
Exalead Bot Disallow	18	0,14
Google News Bot Disallow	15	0,12
DuckDuckGo Bot Disallow	15	0,12
Google Video Bot Disallow	13	0,1
Pipl Disallow	13	0,1
360 Bot Disallow	12	0,1
Google Favicon Disallow	12	0,1
Google API Bot Disallow	10	0,08
Google DuplexWeb Disallow	10	0,08
BuiltWithBot Disallow	4	0,03
Youdao Bot Disallow	1	0,01