Robots.txt Nutzungsverteilung In Wallis und Futuna

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	92	15,65
Baidu Bot Disallow	60	10,2
Googlebot Disallow	59	10,03
Yandex Bot Disallow	45	7,65
Microsoft BingBot Disallow	45	7,65
Alexa Bot Disallow	45	7,65
Yahoo Slurp Disallow	45	7,65
Sogou Bot Disallow	43	7,31
Soso Bot Disallow	43	7,31
Google Media Partners Disallow	16 ^*	2,72
Ahrefs Bot Disallow	10	1,7
Microsoft MSNBot Disallow	8	1,36
Google Image Bot Disallow	6	1,02
Google AdsBot Disallow	5	0,85
Semrush Bot Disallow	5	0,85
Google AdsBot Mobile Disallow	4	0,68
GPTBot Disallow	4	0,68
Common Crawl Bot Disallow	4	0,68
Majestic Bot Disallow	4	0,68
Google AdsBot Mobile Apps Disallow	3	0,51
ByteDance Bytespider Disallow	3	0,51
Exalead Bot Disallow	3	0,51
Google News Bot Disallow	3	0,51
Google Video Bot Disallow	3	0,51
DuckDuckGo Bot Disallow	3	0,51
Google Favicon Disallow	3	0,51
Google API Bot Disallow	3	0,51
Google DuplexWeb Disallow	3	0,51
AppleBot Disallow	2	0,34
Anthropic Claude Bot Disallow	2	0,34
Google Bard Disallow	2	0,34
Facebook Bot Disallow	2	0,34
Apple AI Bot Disallow	2	0,34
Webz Omgili Disallow	2	0,34
Cohere AI Disallow	1	0,17
Perplexity	1	0,17
360 Bot Disallow	1	0,17
BuiltWithBot Disallow	1	0,17
Diffbot Disallow	1	0,17
Youdao Bot Disallow	1	0,17