Robots.txt Usage Distribution in Saudi Arabia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Ahrefs Bot Disallow	4,504	10.27
Common Crawl Bot Disallow	4,154	61.4
Majestic Bot Disallow	4,017	9.16
Exalead Bot Disallow	3,898	8.89
360 Bot Disallow	3,897	8.88
BuiltWithBot Disallow	3,886	8.86
Sogou Bot Disallow	3,632	8.28
Soso Bot Disallow	3,630	8.27
Google AdsBot Disallow	2,199	5.01
Googlebot Disallow	1,903	4.34
Google AdsBot Mobile Disallow	1,700	3.88
Robots Disallow	1,324	3.02
Baidu Bot Disallow	405	0.92
GPTBot Disallow	389	5.75
Google Bard Disallow	348	5.14
Some Robots Disallow	343	0.78
Anthropic Claude Bot Disallow	324	4.79
Facebook Bot Disallow	321	4.74
AppleBot Disallow	317	0.72
Google AdsBot Mobile Apps Disallow	316	0.72
Cohere AI Disallow	305	4.51
Apple AI Bot Disallow	304	4.49
Perplexity	304	4.49
Yandex Bot Disallow	282	0.64
Microsoft BingBot Disallow	231	0.53
Alexa Bot Disallow	229	3.38
Yahoo Slurp Disallow	224	0.51
Semrush Bot Disallow	191	0.44
Google Media Partners Disallow	81 ^*	0.18
Microsoft MSNBot Disallow	45	0.1
Google Image Bot Disallow	41	0.09
ByteDance Bytespider Disallow	29	0.43
Webz Omgili Disallow	17	0.25
Google News Bot Disallow	17	0.04
Google Video Bot Disallow	14	0.03
DuckDuckGo Bot Disallow	12	0.03
Youdao Bot Disallow	7	0.02
Diffbot Disallow	6	0.09
Google Favicon Disallow	6	0.01
Google API Bot Disallow	6	0.01
Google DuplexWeb Disallow	5	0.01
Pipl Disallow	3	0.01
YOU Disallow	1	0.01
Google FeedFetcher Disallow	1	0