Robots.txt Usage Distribution in Iran

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	5,417	13.64
Ahrefs Bot Disallow	4,750	11.96
Semrush Bot Disallow	4,297	10.82
Microsoft BingBot Disallow	2,479	6.24
Common Crawl Bot Disallow	2,444	6.15
Exalead Bot Disallow	2,350	5.92
Microsoft MSNBot Disallow	2,129	5.36
Googlebot Disallow	1,554	3.91
Robots Disallow	1,534	3.86
Yandex Bot Disallow	1,491	3.75
Sogou Bot Disallow	1,404	3.54
Baidu Bot Disallow	1,321	3.33
Soso Bot Disallow	1,022	2.57
Yahoo Slurp Disallow	996	2.51
ByteDance Bytespider Disallow	826	2.08
Some Robots Disallow	705	1.78
360 Bot Disallow	693	1.74
BuiltWithBot Disallow	655	1.65
GPTBot Disallow	630	1.59
Alexa Bot Disallow	540	1.36
AppleBot Disallow	380	0.96
DuckDuckGo Bot Disallow	310	0.78
Google Media Partners Disallow	276 ^*	0.69
Google Image Bot Disallow	204	0.51
Google Bard Disallow	167	0.42
Anthropic Claude Bot Disallow	139	0.35
Google News Bot Disallow	132	0.33
Google Video Bot Disallow	131	0.33
Google AdsBot Disallow	117	0.29
Google AdsBot Mobile Disallow	82	0.21
Perplexity	72	0.18
Apple AI Bot Disallow	70	0.18
Facebook Bot Disallow	67	0.17
Cohere AI Disallow	65	0.16
Google AdsBot Mobile Apps Disallow	56	0.14
Webz Omgili Disallow	53	0.13
Google Favicon Disallow	30	0.08
Google API Bot Disallow	30	0.08
Google DuplexWeb Disallow	30	0.08
Diffbot Disallow	27	0.07
Pipl Disallow	20	0.05
YOU Disallow	17	0.04
Youdao Bot Disallow	2	0.01