Robots.txt Usage Distribution in Jordan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	753	15.64
Google AdsBot Mobile Disallow	550	11.43
Googlebot Disallow	538	11.18
Ahrefs Bot Disallow	253	5.26
Robots Disallow	247	5.13
Common Crawl Bot Disallow	170	3.53
GPTBot Disallow	168	3.49
Google AdsBot Mobile Apps Disallow	163	3.39
Anthropic Claude Bot Disallow	154	3.2
Google Bard Disallow	154	3.2
Facebook Bot Disallow	153	3.18
AppleBot Disallow	150	3.12
Apple AI Bot Disallow	150	3.12
Cohere AI Disallow	150	3.12
Perplexity	149	3.1
Baidu Bot Disallow	138	2.87
Some Robots Disallow	117	2.43
Yandex Bot Disallow	108	2.24
Yahoo Slurp Disallow	82	1.7
Microsoft BingBot Disallow	75	1.56
Alexa Bot Disallow	74	1.54
Sogou Bot Disallow	64	1.33
Soso Bot Disallow	63	1.31
Semrush Bot Disallow	46	0.96
Majestic Bot Disallow	44	0.91
Microsoft MSNBot Disallow	21	0.44
Google Media Partners Disallow	16 ^*	0.33
Google Image Bot Disallow	15	0.31
Exalead Bot Disallow	8	0.17
ByteDance Bytespider Disallow	6	0.12
360 Bot Disallow	5	0.1
Google News Bot Disallow	5	0.1
Google Video Bot Disallow	5	0.1
BuiltWithBot Disallow	4	0.08
Youdao Bot Disallow	4	0.08
Webz Omgili Disallow	3	0.06
DuckDuckGo Bot Disallow	3	0.06
Google Favicon Disallow	2	0.04
Google API Bot Disallow	2	0.04
Google DuplexWeb Disallow	2	0.04