Robots.txt Répartition des utilisations sur l'île de Man

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Googlebot Disallow	2 082	10,34
Baidu Bot Disallow	2 008	9,97
Sogou Bot Disallow	1 930	9,59
Yandex Bot Disallow	1 702	8,45
Microsoft BingBot Disallow	1 260	6,26
Alexa Bot Disallow	1 202	31,25
Yahoo Slurp Disallow	1 197	5,95
Soso Bot Disallow	1 190	5,91
Some Robots Disallow	1 053	5,23
Robots Disallow	767	3,81
Google AdsBot Disallow	766	3,8
Ahrefs Bot Disallow	740	3,68
Anthropic Claude Bot Disallow	589	15,31
Semrush Bot Disallow	565	2,81
Majestic Bot Disallow	558	2,77
GPTBot Disallow	536	13,94
Common Crawl Bot Disallow	509	13,23
Google Bard Disallow	497	12,92
Google Media Partners Disallow	133 ^*	0,66
Google AdsBot Mobile Disallow	114	0,57
Facebook Bot Disallow	97	2,52
AppleBot Disallow	92	0,46
Apple AI Bot Disallow	88	2,29
Cohere AI Disallow	63	1,64
Perplexity	60	1,56
Webz Omgili Disallow	55	1,43
Google AdsBot Mobile Apps Disallow	48	0,24
ByteDance Bytespider Disallow	35	0,91
Microsoft MSNBot Disallow	27	0,13
Exalead Bot Disallow	26	0,13
360 Bot Disallow	25	0,12
Diffbot Disallow	20	0,52
Google Image Bot Disallow	17	0,08
YOU Disallow	15	0,39
DuckDuckGo Bot Disallow	10	0,05
Google News Bot Disallow	8	0,04
Google Video Bot Disallow	8	0,04
Google Favicon Disallow	8	0,04
Google API Bot Disallow	8	0,04
Google DuplexWeb Disallow	8	0,04
Pipl Disallow	7	0,03
BuiltWithBot Disallow	4	0,02
Google FeedFetcher Disallow	4	0,02
LLMS Text	1	0