Robots.txt Distribución de uso en Ir

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Robots Disallow	108	16,8
Googlebot Disallow	87	13,53
Baidu Bot Disallow	49	7,62
Sogou Bot Disallow	39	6,07
Yandex Bot Disallow	36	5,6
Alexa Bot Disallow	34	5,29
Yahoo Slurp Disallow	33	5,13
Soso Bot Disallow	33	5,13
Google Media Partners Disallow	32 ^*	4,98
Some Robots Disallow	32	4,98
Microsoft BingBot Disallow	32	4,98
Ahrefs Bot Disallow	21	3,27
GPTBot Disallow	12	1,87
Common Crawl Bot Disallow	10	1,56
AppleBot Disallow	8	1,24
Anthropic Claude Bot Disallow	8	1,24
Google Bard Disallow	8	1,24
Facebook Bot Disallow	8	1,24
Apple AI Bot Disallow	8	1,24
Perplexity	8	1,24
Cohere AI Disallow	7	1,09
Google AdsBot Mobile Apps Disallow	7	1,09
Majestic Bot Disallow	7	1,09
Semrush Bot Disallow	7	1,09
ByteDance Bytespider Disallow	3	0,47
Exalead Bot Disallow	1	0,16
Microsoft MSNBot Disallow	1	0,16
Webz Omgili Disallow	1	0,16
360 Bot Disallow	1	0,16
Google Image Bot Disallow	1	0,16
BuiltWithBot Disallow	1	0,16