Robots.txt Répartition de l'utilisation dans les 100 000 meilleurs sites

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Robots Disallow	28 678	28,68
Googlebot Disallow	9 374	9,37
Some Robots Disallow	8 556	8,56
Ahrefs Bot Disallow	8 507	8,51
GPTBot Disallow	7 708	7,71
Yandex Bot Disallow	6 244	6,24
Semrush Bot Disallow	5 911	5,91
Common Crawl Bot Disallow	5 372	5,37
Google AdsBot Disallow	5 247	5,25
Majestic Bot Disallow	5 019	5,02
Baidu Bot Disallow	4 589	4,59
Anthropic Claude Bot Disallow	4 165	4,16
Microsoft BingBot Disallow	3 976	3,98
AppleBot Disallow	3 907	3,91
Google Bard Disallow	3 882	3,88
Facebook Bot Disallow	2 912	2,91
Alexa Bot Disallow	2 778	2,78
ByteDance Bytespider Disallow	2 730	2,73
Yahoo Slurp Disallow	2 589	2,59
Perplexity	2 509	2,51
Webz Omgili Disallow	2 479	2,48
Google Media Partners Disallow	2 441	2,44
Microsoft MSNBot Disallow	2 337	2,34
Apple AI Bot Disallow	2 255	2,26
Cohere AI Disallow	2 151	2,15
Google Image Bot Disallow	1 992	1,99
Google AdsBot Mobile Disallow	1 884	1,88
Sogou Bot Disallow	1 772	1,77
Exalead Bot Disallow	1 604	1,6
Google News Bot Disallow	1 402	1,4
Soso Bot Disallow	1 328	1,33
Diffbot Disallow	1 293	1,29
Google AdsBot Mobile Apps Disallow	1 224	1,22
DuckDuckGo Bot Disallow	1 058	1,06
360 Bot Disallow	999	1
Google Video Bot Disallow	903	0,9
YOU Disallow	793	0,79
Google Favicon Disallow	718	0,72
Google API Bot Disallow	704	0,7
Google DuplexWeb Disallow	663	0,66
BuiltWithBot Disallow	505	0,5
Pipl Disallow	417	0,42
Youdao Bot Disallow	194	0,19
Google FeedFetcher Disallow	32	0,03
Google Read Aloud Disallow	20	0,02
Google Web Light Disallow	6	0,01
Google BIG Bench Disallow	1	0