Robots.txt Distribución de uso en Irán

Distribución para sitios web que utilizan Robots.txt tecnologías

Tecnología	Sitios web	%
Majestic Bot Disallow	5.654	13,6
Ahrefs Bot Disallow	4.965	11,94
Semrush Bot Disallow	4.495	10,81
Microsoft BingBot Disallow	2.526	6,07
Common Crawl Bot Disallow	2.457	5,91
Exalead Bot Disallow	2.361	5,68
Microsoft MSNBot Disallow	2.163	5,2
Robots Disallow	2.003	4,82
Yandex Bot Disallow	1.738	4,18
Googlebot Disallow	1.575	3,79
Sogou Bot Disallow	1.337	3,21
Baidu Bot Disallow	1.312	3,15
Soso Bot Disallow	1.037	2,49
Yahoo Slurp Disallow	1.034	2,49
ByteDance Bytespider Disallow	892	2,14
Some Robots Disallow	878	2,11
360 Bot Disallow	699	1,68
BuiltWithBot Disallow	657	1,58
GPTBot Disallow	645	1,55
Alexa Bot Disallow	558	1,34
AppleBot Disallow	386	0,93
DuckDuckGo Bot Disallow	325	0,78
Google Media Partners Disallow	287 ^*	0,69
Google Image Bot Disallow	218	0,52
Google Bard Disallow	170	0,41
Anthropic Claude Bot Disallow	143	0,34
Google News Bot Disallow	142	0,34
Google Video Bot Disallow	141	0,34
Google AdsBot Disallow	119	0,29
Google AdsBot Mobile Disallow	84	0,2
Perplexity	75	0,18
Apple AI Bot Disallow	73	0,18
Facebook Bot Disallow	70	0,17
Cohere AI Disallow	68	0,16
Google AdsBot Mobile Apps Disallow	65	0,16
Webz Omgili Disallow	54	0,13
Google Favicon Disallow	37	0,09
Google API Bot Disallow	37	0,09
Google DuplexWeb Disallow	37	0,09
Diffbot Disallow	28	0,07
Pipl Disallow	21	0,05
YOU Disallow	17	0,04
Youdao Bot Disallow	4	0,01