Robots.txt Nutzungsverteilung in den Niederlanden

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	84.134	10,18
Google AdsBot Disallow	73.447	8,88
Robots Disallow	58.908	7,12
Baidu Bot Disallow	52.811	6,39
Ahrefs Bot Disallow	50.550	6,11
Google AdsBot Mobile Disallow	44.421	5,37
Sogou Bot Disallow	42.557	5,15
Majestic Bot Disallow	34.943	4,23
Yandex Bot Disallow	32.431	3,92
Some Robots Disallow	30.879	3,73
Semrush Bot Disallow	22.855	2,76
Microsoft BingBot Disallow	21.744	2,63
Alexa Bot Disallow	21.023	11,39
GPTBot Disallow	20.687	11,21
Yahoo Slurp Disallow	20.659	2,5
AppleBot Disallow	19.831	2,4
Common Crawl Bot Disallow	19.218	10,41
Soso Bot Disallow	18.052	2,18
Anthropic Claude Bot Disallow	17.765	9,62
Google Bard Disallow	17.234	9,34
Facebook Bot Disallow	16.055	8,7
Apple AI Bot Disallow	15.539	8,42
Perplexity	14.599	7,91
Cohere AI Disallow	14.239	7,71
Google AdsBot Mobile Apps Disallow	12.526	1,52
ByteDance Bytespider Disallow	9.142	4,95
Exalead Bot Disallow	7.209	0,87
Microsoft MSNBot Disallow	5.178	0,63
Google Image Bot Disallow	5.096	0,62
Google Media Partners Disallow	4.232 ^*	0,51
Webz Omgili Disallow	3.974	2,15
Diffbot Disallow	2.838	1,54
YOU Disallow	2.158	1,17
360 Bot Disallow	1.874	0,23
DuckDuckGo Bot Disallow	1.643	0,2
Google News Bot Disallow	1.111	0,13
BuiltWithBot Disallow	1.061	0,13
Google Video Bot Disallow	1.021	0,12
Google Favicon Disallow	966	0,12
Google API Bot Disallow	884	0,11
Google DuplexWeb Disallow	880	0,11
Pipl Disallow	345	0,04
Youdao Bot Disallow	47	0,01
Google FeedFetcher Disallow	14	0
Google Read Aloud Disallow	4	0
LLMS Text	4	0
Google Web Light Disallow	3	0