Robots.txt Distribución de uso en santa elena

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Googlebot Disallow	1.642	16,6
Baidu Bot Disallow	1.534	15,51
Some Robots Disallow	1.262	12,76
Robots Disallow	992	10,03
Yandex Bot Disallow	453	4,58
Microsoft BingBot Disallow	437	4,42
Alexa Bot Disallow	428	32,95
Yahoo Slurp Disallow	428	4,33
Sogou Bot Disallow	414	4,18
Soso Bot Disallow	406	4,1
Google AdsBot Disallow	187	1,89
Ahrefs Bot Disallow	185	1,87
GPTBot Disallow	176	13,55
Google Media Partners Disallow	145 ^*	1,47
Majestic Bot Disallow	114	1,15
Semrush Bot Disallow	100	1,01
Common Crawl Bot Disallow	98	7,54
Google Bard Disallow	98	7,54
Google AdsBot Mobile Disallow	86	0,87
Anthropic Claude Bot Disallow	81	6,24
AppleBot Disallow	75	0,76
Facebook Bot Disallow	75	5,77
Apple AI Bot Disallow	62	4,77
Webz Omgili Disallow	54	4,16
Perplexity	49	3,77
Cohere AI Disallow	48	3,7
ByteDance Bytespider Disallow	36	2,77
Google AdsBot Mobile Apps Disallow	33	0,33
Microsoft MSNBot Disallow	28	0,28
Google Image Bot Disallow	25	0,25
Diffbot Disallow	23	1,77
YOU Disallow	17	1,31
Exalead Bot Disallow	15	0,15
Google News Bot Disallow	14	0,14
DuckDuckGo Bot Disallow	13	0,13
Google Video Bot Disallow	12	0,12
Google Favicon Disallow	11	0,11
Google API Bot Disallow	9	0,09
Google DuplexWeb Disallow	9	0,09
Pipl Disallow	9	0,09
360 Bot Disallow	7	0,07
BuiltWithBot Disallow	2	0,02
Youdao Bot Disallow	1	0,01