Robots.txt Nutzungsverteilung In Australien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	368.112	11,98
Google AdsBot Mobile Disallow	264.032	8,6
Googlebot Disallow	239.571	7,8
Baidu Bot Disallow	162.809	5,3
GPTBot Disallow	136.792	4,45
Common Crawl Bot Disallow	134.271	4,37
Microsoft BingBot Disallow	133.541	4,35
Google AdsBot Mobile Apps Disallow	131.060	4,27
Anthropic Claude Bot Disallow	124.625	4,06
Google Bard Disallow	124.453	4,05
Facebook Bot Disallow	123.865	4,03
AppleBot Disallow	122.328	3,98
Cohere AI Disallow	122.166	3,98
Apple AI Bot Disallow	120.658	3,93
Perplexity	119.795	3,9
Ahrefs Bot Disallow	114.261	3,72
Yandex Bot Disallow	78.349	2,55
Alexa Bot Disallow	77.059	2,51
Yahoo Slurp Disallow	75.621	2,46
Sogou Bot Disallow	74.331	2,42
Soso Bot Disallow	72.639	2,36
Some Robots Disallow	51.846	1,69
Robots Disallow	45.613	1,48
Majestic Bot Disallow	14.457	0,47
Semrush Bot Disallow	11.487	0,37
Microsoft MSNBot Disallow	4.801	0,16
ByteDance Bytespider Disallow	3.871	0,13
Exalead Bot Disallow	3.716	0,12
Webz Omgili Disallow	2.966	0,1
Google Image Bot Disallow	2.875	0,09
360 Bot Disallow	2.084	0,07
Google Media Partners Disallow	1.858 ^*	0,06
Google News Bot Disallow	1.026	0,03
Diffbot Disallow	1.018	0,03
Google Video Bot Disallow	919	0,03
Pipl Disallow	579	0,02
DuckDuckGo Bot Disallow	567	0,02
BuiltWithBot Disallow	481	0,02
Google Favicon Disallow	473	0,02
Google API Bot Disallow	273	0,01
Google DuplexWeb Disallow	263	0,01
YOU Disallow	207	0,01
Youdao Bot Disallow	182	0,01
Google FeedFetcher Disallow	15	0
Google Read Aloud Disallow	1	0
Google Web Light Disallow	1	0