Robots.txt Nutzungsverteilung In Vereinigte Staaten

Verteilung für Websites mit Robots.txt Technologien

Anheften Robots.txt Nutzungsverteilung In Vereinigte Staaten

Technologie	Websites	%
Common Crawl Bot Disallow	2.548.584	5,03
GPTBot Disallow	2.543.793	5,02
Yandex Bot Disallow	2.530.642	4,99
Ahrefs Bot Disallow	2.493.519	4,92
Anthropic Claude Bot Disallow	2.384.960	4,7
AppleBot Disallow	2.375.407	4,69
Microsoft BingBot Disallow	2.370.443	4,68
Alexa Bot Disallow	2.362.221	4,66
Google Bard Disallow	2.338.402	4,61
Robots Disallow	2.326.691	4,59
Facebook Bot Disallow	2.317.026	4,57
Apple AI Bot Disallow	2.282.881	4,5
Yahoo Slurp Disallow	2.242.435	4,42
Cohere AI Disallow	2.197.930	4,34
Perplexity	2.190.749	4,32
Google AdsBot Disallow	2.138.149	4,22
Soso Bot Disallow	2.128.968	4,2
Google AdsBot Mobile Apps Disallow	2.124.082	4,19
Google AdsBot Mobile Disallow	1.712.830	3,38
Googlebot Disallow	1.441.919	2,84
Majestic Bot Disallow	1.071.704	2,11
Semrush Bot Disallow	944.141	1,86
Baidu Bot Disallow	628.557	1,24
Some Robots Disallow	553.846	1,09
ByteDance Bytespider Disallow	325.099	0,64
Google Media Partners Disallow	280.113 ^*	0,55
Sogou Bot Disallow	256.632	0,51
Exalead Bot Disallow	244.065	0,48
Microsoft MSNBot Disallow	242.094	0,48
360 Bot Disallow	227.257	0,45
Webz Omgili Disallow	214.453	0,42
Google Image Bot Disallow	193.857	0,38
Diffbot Disallow	90.434	0,18
BuiltWithBot Disallow	88.432	0,17
Google News Bot Disallow	48.387	0,1
Google Video Bot Disallow	48.268	0,1
Youdao Bot Disallow	47.032	0,09
Google Favicon Disallow	38.510	0,08
DuckDuckGo Bot Disallow	35.134	0,07
YOU Disallow	27.340	0,05
Google API Bot Disallow	11.483	0,02
Google DuplexWeb Disallow	11.016	0,02
Pipl Disallow	10.871	0,02
Google FeedFetcher Disallow	2.022	0
Google Read Aloud Disallow	217	0
Google Web Light Disallow	50	0
Google BIG Bench Disallow	18	0