Robots.txt Répartition des utilisations en France

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Googlebot Disallow	158 685	13,48
Google AdsBot Disallow	158 357	13,45
Google AdsBot Mobile Disallow	115 770	9,83
Robots Disallow	90 918	7,72
Ahrefs Bot Disallow	63 814	5,42
Baidu Bot Disallow	49 917	4,24
Sogou Bot Disallow	44 202	3,75
Yandex Bot Disallow	36 690	3,12
Microsoft BingBot Disallow	31 728	2,69
Yahoo Slurp Disallow	30 975	2,63
Some Robots Disallow	27 465	2,33
Alexa Bot Disallow	27 060	13,45
Soso Bot Disallow	27 053	2,3
GPTBot Disallow	24 342	12,1
Semrush Bot Disallow	23 927	2,03
Majestic Bot Disallow	22 286	1,89
Common Crawl Bot Disallow	21 522	10,7
Anthropic Claude Bot Disallow	19 721	9,8
Google Bard Disallow	18 387	9,14
AppleBot Disallow	18 334	1,56
Facebook Bot Disallow	18 145	9,02
Google AdsBot Mobile Apps Disallow	17 996	1,53
Perplexity	16 793	8,35
Cohere AI Disallow	15 975	7,94
Apple AI Bot Disallow	15 542	7,73
Google Media Partners Disallow	11 145 ^*	0,95
Microsoft MSNBot Disallow	7 621	0,65
Google Image Bot Disallow	7 215	0,61
DuckDuckGo Bot Disallow	6 060	0,51
ByteDance Bytespider Disallow	5 995	2,98
Webz Omgili Disallow	4 916	2,44
Youdao Bot Disallow	4 847	0,41
Google News Bot Disallow	4 775	0,41
Google Video Bot Disallow	4 731	0,4
Exalead Bot Disallow	4 679	0,4
Google Favicon Disallow	4 014	0,34
Google API Bot Disallow	3 977	0,34
Google DuplexWeb Disallow	3 961	0,34
Diffbot Disallow	2 744	1,36
YOU Disallow	2 559	1,27
360 Bot Disallow	1 523	0,13
BuiltWithBot Disallow	532	0,05
Pipl Disallow	422	0,04
Google FeedFetcher Disallow	15	0
Google Read Aloud Disallow	13	0
LLMS Text	13	0
Google Web Light Disallow	2	0