Robots.txt Nutzungsverteilung in Benin

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	136	30,49
Googlebot Disallow	56	12,56
Baidu Bot Disallow	28	6,28
Ahrefs Bot Disallow	25	5,61
Yandex Bot Disallow	24	5,38
Sogou Bot Disallow	15	3,36
Google Media Partners Disallow	15 ^*	3,36
Microsoft BingBot Disallow	14	3,14
Alexa Bot Disallow	13	2,91
Majestic Bot Disallow	13	2,91
Semrush Bot Disallow	13	2,91
Yahoo Slurp Disallow	12	2,69
Soso Bot Disallow	12	2,69
GPTBot Disallow	8	1,79
Google Bard Disallow	7	1,57
Common Crawl Bot Disallow	6	1,35
Google AdsBot Mobile Apps Disallow	6	1,35
AppleBot Disallow	5	1,12
Anthropic Claude Bot Disallow	5	1,12
Facebook Bot Disallow	5	1,12
Cohere AI Disallow	5	1,12
Apple AI Bot Disallow	5	1,12
Perplexity	5	1,12
Google Image Bot Disallow	2	0,45
Exalead Bot Disallow	1	0,22
Microsoft MSNBot Disallow	1	0,22
360 Bot Disallow	1	0,22
Google News Bot Disallow	1	0,22
Google Video Bot Disallow	1	0,22
DuckDuckGo Bot Disallow	1	0,22
Youdao Bot Disallow	1	0,22
Google Favicon Disallow	1	0,22
Google API Bot Disallow	1	0,22
Google DuplexWeb Disallow	1	0,22
Pipl Disallow	1	0,22