Robots.txt Usage Distribution in Saudi Arabia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Ahrefs Bot Disallow	4,544	10.02
Common Crawl Bot Disallow	4,199	60.24
Soso Bot Disallow	4,121	9.08
Sogou Bot Disallow	4,035	8.89
Majestic Bot Disallow	4,012	8.84
Exalead Bot Disallow	3,910	8.62
360 Bot Disallow	3,904	8.61
BuiltWithBot Disallow	3,896	8.59
Google AdsBot Disallow	2,149	4.74
Googlebot Disallow	1,888	4.16
Google AdsBot Mobile Disallow	1,651	3.64
Robots Disallow	1,606	3.54
Baidu Bot Disallow	413	0.91
GPTBot Disallow	384	5.51
Some Robots Disallow	377	0.83
Google Bard Disallow	358	5.14
Anthropic Claude Bot Disallow	342	4.91
AppleBot Disallow	341	0.75
Facebook Bot Disallow	337	4.84
Perplexity	330	4.73
Apple AI Bot Disallow	326	4.68
Cohere AI Disallow	320	4.59
Google AdsBot Mobile Apps Disallow	317	0.7
Yandex Bot Disallow	303	0.67
Alexa Bot Disallow	265	3.8
Microsoft BingBot Disallow	264	0.58
Yahoo Slurp Disallow	256	0.56
Semrush Bot Disallow	174	0.38
Google Media Partners Disallow	135 ^*	0.3
ByteDance Bytespider Disallow	38	0.55
Google Image Bot Disallow	37	0.08
Microsoft MSNBot Disallow	36	0.08
Webz Omgili Disallow	20	0.29
Google News Bot Disallow	15	0.03
Google Video Bot Disallow	12	0.03
DuckDuckGo Bot Disallow	12	0.03
Diffbot Disallow	8	0.11
Google Favicon Disallow	7	0.02
Google API Bot Disallow	7	0.02
Google DuplexWeb Disallow	6	0.01
Youdao Bot Disallow	5	0.01
YOU Disallow	4	0.06
Pipl Disallow	3	0.01
Google FeedFetcher Disallow	1	0