Robots.txt Répartition de l'utilisation dans Mauritanie

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Robots Disallow	26	18,71
Yandex Bot Disallow	12	8,63
Alexa Bot Disallow	11	7,91
Microsoft BingBot Disallow	9	6,47
Ahrefs Bot Disallow	9	6,47
Yahoo Slurp Disallow	9	6,47
Sogou Bot Disallow	8	5,76
Soso Bot Disallow	8	5,76
GPTBot Disallow	4	2,88
Google AdsBot Mobile Apps Disallow	4	2,88
Majestic Bot Disallow	4	2,88
Common Crawl Bot Disallow	3	2,16
AppleBot Disallow	3	2,16
Anthropic Claude Bot Disallow	3	2,16
Google Bard Disallow	3	2,16
Facebook Bot Disallow	3	2,16
Apple AI Bot Disallow	3	2,16
Cohere AI Disallow	3	2,16
Perplexity	3	2,16
Semrush Bot Disallow	3	2,16
Microsoft MSNBot Disallow	1	0,72
Google Image Bot Disallow	1	0,72
Google News Bot Disallow	1	0,72
Google Video Bot Disallow	1	0,72
DuckDuckGo Bot Disallow	1	0,72
Google Favicon Disallow	1	0,72
Google API Bot Disallow	1	0,72
Google DuplexWeb Disallow	1	0,72