Robots.txt Répartition des utilisations au Rwanda

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Common Crawl Bot Disallow	198	8,04
Sogou Bot Disallow	191	7,75
Googlebot Disallow	189	7,67
Ahrefs Bot Disallow	181	7,35
Soso Bot Disallow	166	6,74
Majestic Bot Disallow	166	6,74
Semrush Bot Disallow	163	6,62
Exalead Bot Disallow	149	6,05
360 Bot Disallow	144	5,85
BuiltWithBot Disallow	144	5,85
Robots Disallow	114	4,63
GPTBot Disallow	62	2,52
AppleBot Disallow	58	2,35
Perplexity	53	2,15
Google AdsBot Mobile Apps Disallow	53	2,15
Anthropic Claude Bot Disallow	52	2,11
Google Bard Disallow	52	2,11
Facebook Bot Disallow	52	2,11
Apple AI Bot Disallow	52	2,11
Cohere AI Disallow	51	2,07
Yandex Bot Disallow	43	1,75
Microsoft BingBot Disallow	30	1,22
Yahoo Slurp Disallow	30	1,22
Alexa Bot Disallow	27	1,1
Google Media Partners Disallow	15 ^*	0,61
Microsoft MSNBot Disallow	6	0,24
Google Image Bot Disallow	4	0,16
DuckDuckGo Bot Disallow	3	0,12
ByteDance Bytespider Disallow	2	0,08
Webz Omgili Disallow	2	0,08
Google News Bot Disallow	2	0,08
Google Video Bot Disallow	2	0,08
Google Favicon Disallow	2	0,08
Google API Bot Disallow	2	0,08
Google DuplexWeb Disallow	2	0,08
YOU Disallow	1	0,04