Robots.txt Usage Distribution in Iran

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	5,250	13.73
Ahrefs Bot Disallow	4,610	12.06
Semrush Bot Disallow	4,190	10.96
Common Crawl Bot Disallow	2,621	6.85
Exalead Bot Disallow	2,566	6.71
Microsoft BingBot Disallow	2,317	6.06
Microsoft MSNBot Disallow	2,188	5.72
Robots Disallow	1,508	3.94
Googlebot Disallow	1,460	3.82
Yandex Bot Disallow	1,431	3.74
Baidu Bot Disallow	1,283	3.36
Sogou Bot Disallow	1,207	3.16
Soso Bot Disallow	1,061	2.77
Yahoo Slurp Disallow	910	2.38
360 Bot Disallow	814	2.13
BuiltWithBot Disallow	786	2.06
GPTBot Disallow	616	1.61
Some Robots Disallow	585	1.53
Alexa Bot Disallow	505	1.32
AppleBot Disallow	374	0.98
DuckDuckGo Bot Disallow	292	0.76
Google Image Bot Disallow	198	0.52
Google Bard Disallow	165	0.43
Anthropic Claude Bot Disallow	139	0.36
Google AdsBot Disallow	133	0.35
Google News Bot Disallow	125	0.33
Google Video Bot Disallow	122	0.32
Google AdsBot Mobile Disallow	97	0.25
Google Media Partners Disallow	81 ^*	0.21
ByteDance Bytespider Disallow	80	0.21
Cohere AI Disallow	71	0.19
Perplexity	69	0.18
Facebook Bot Disallow	66	0.17
Google AdsBot Mobile Apps Disallow	66	0.17
Apple AI Bot Disallow	61	0.16
Webz Omgili Disallow	48	0.13
Google Favicon Disallow	28	0.07
Google API Bot Disallow	28	0.07
Google DuplexWeb Disallow	28	0.07
Diffbot Disallow	22	0.06
Pipl Disallow	21	0.05
YOU Disallow	13	0.03
Youdao Bot Disallow	3	0.01