Robots.txt Nutzungsverteilung im gesamten Internet

Verteilung für Websites mit Robots.txt Technologien

Anheften Robots.txt Nutzungsverteilung im gesamten Internet

Technologie	Websites	%
Some Robots Disallow	39.138.226	20,7
Googlebot Disallow	17.586.304	9,3
Baidu Bot Disallow	12.526.098	6,62
Google Media Partners Disallow	11.346.773	6
Google AdsBot Disallow	11.034.816	5,84
Sogou Bot Disallow	9.096.820	4,81
Google AdsBot Mobile Disallow	8.178.514	4,32
Yandex Bot Disallow	6.130.801	3,24
Microsoft BingBot Disallow	5.819.207	3,08
Alexa Bot Disallow	5.517.664	2,92
Yahoo Slurp Disallow	5.311.466	2,81
Robots Disallow	5.158.350	2,73
Soso Bot Disallow	5.107.738	2,7
Ahrefs Bot Disallow	5.086.908	2,69
Common Crawl Bot Disallow	3.667.845	1,94
GPTBot Disallow	3.659.619	1,94
AppleBot Disallow	3.541.566	1,87
Anthropic Claude Bot Disallow	3.450.206	1,82
Google Bard Disallow	3.295.919	1,74
Facebook Bot Disallow	3.263.341	1,73
Apple AI Bot Disallow	3.123.305	1,65
Cohere AI Disallow	3.088.048	1,63
Perplexity	3.035.891	1,61
Google AdsBot Mobile Apps Disallow	2.897.406	1,53
Majestic Bot Disallow	2.601.232	1,38
Semrush Bot Disallow	2.300.807	1,22
ByteDance Bytespider Disallow	849.886	0,45
Exalead Bot Disallow	537.715	0,28
Microsoft MSNBot Disallow	481.155	0,25
Webz Omgili Disallow	455.931	0,24
360 Bot Disallow	373.186	0,2
Google Image Bot Disallow	372.087	0,2
Diffbot Disallow	190.672	0,1
BuiltWithBot Disallow	186.760	0,1
Google News Bot Disallow	105.299	0,06
Google Video Bot Disallow	103.305	0,05
YOU Disallow	98.654	0,05
DuckDuckGo Bot Disallow	97.289	0,05
Google Favicon Disallow	86.203	0,05
Youdao Bot Disallow	75.051	0,04
Google API Bot Disallow	55.202	0,03
Google DuplexWeb Disallow	54.139	0,03
Pipl Disallow	20.462	0,01
Google FeedFetcher Disallow	3.912	0
Google Read Aloud Disallow	440	0
Google Web Light Disallow	108	0
Google BIG Bench Disallow	32	0