Robots.txt Répartition de l'utilisation dans Rwanda

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Common Crawl Bot Disallow	226	8,33
Ahrefs Bot Disallow	199	7,34
Googlebot Disallow	183	6,75
Semrush Bot Disallow	180	6,63
Majestic Bot Disallow	179	6,6
Sogou Bot Disallow	169	6,23
Soso Bot Disallow	168	6,19
Exalead Bot Disallow	155	5,71
360 Bot Disallow	148	5,46
BuiltWithBot Disallow	148	5,46
Robots Disallow	102	3,76
GPTBot Disallow	89	3,28
Google AdsBot Mobile Apps Disallow	78	2,88
AppleBot Disallow	77	2,84
Anthropic Claude Bot Disallow	71	2,62
Google Bard Disallow	71	2,62
Cohere AI Disallow	71	2,62
Perplexity	71	2,62
Facebook Bot Disallow	70	2,58
Apple AI Bot Disallow	70	2,58
Yandex Bot Disallow	54	1,99
Yahoo Slurp Disallow	31	1,14
Microsoft BingBot Disallow	30	1,11
Alexa Bot Disallow	29	1,07
Google Media Partners Disallow	16 ^*	0,59
Google Image Bot Disallow	6	0,22
Microsoft MSNBot Disallow	5	0,18
DuckDuckGo Bot Disallow	3	0,11
Google News Bot Disallow	2	0,07
Google Video Bot Disallow	2	0,07
Google Favicon Disallow	2	0,07
Google API Bot Disallow	2	0,07
Google DuplexWeb Disallow	2	0,07
YOU Disallow	2	0,07
ByteDance Bytespider Disallow	1	0,04
Webz Omgili Disallow	1	0,04