Robots.txt Répartition de l'utilisation dans Porto Rico

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Google AdsBot Disallow	4 234	19,6
Google AdsBot Mobile Disallow	2 447	11,33
Googlebot Disallow	2 423	11,21
Ahrefs Bot Disallow	1 478	6,84
AppleBot Disallow	1 048	4,85
Common Crawl Bot Disallow	674	3,12
GPTBot Disallow	672	3,11
Anthropic Claude Bot Disallow	664	3,07
Google Bard Disallow	661	3,06
Facebook Bot Disallow	659	3,05
Apple AI Bot Disallow	657	3,04
Cohere AI Disallow	647	2,99
Perplexity	647	2,99
Google AdsBot Mobile Apps Disallow	639	2,96
Baidu Bot Disallow	604	2,8
Yandex Bot Disallow	594	2,75
Microsoft BingBot Disallow	528	2,44
Yahoo Slurp Disallow	519	2,4
ByteDance Bytespider Disallow	403	1,87
Google Image Bot Disallow	400	1,85
Robots Disallow	248	1,15
Alexa Bot Disallow	139	0,64
Sogou Bot Disallow	131	0,61
Soso Bot Disallow	121	0,56
Some Robots Disallow	117	0,54
Majestic Bot Disallow	73	0,34
Semrush Bot Disallow	68	0,31
Webz Omgili Disallow	27	0,12
Microsoft MSNBot Disallow	23	0,11
Google Media Partners Disallow	17 ^*	0,08
Exalead Bot Disallow	15	0,07
YOU Disallow	9	0,04
Diffbot Disallow	4	0,02
360 Bot Disallow	3	0,01
DuckDuckGo Bot Disallow	3	0,01
BuiltWithBot Disallow	2	0,01
Pipl Disallow	2	0,01
Google News Bot Disallow	1	0
Google Video Bot Disallow	1	0
Google Favicon Disallow	1	0
Google API Bot Disallow	1	0
Google DuplexWeb Disallow	1	0