Robots.txt Usage Distribution in Qatar

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	767	14.01
Googlebot Disallow	623	11.38
Google AdsBot Mobile Disallow	447	8.17
Robots Disallow	430	7.86
Ahrefs Bot Disallow	370	6.76
Baidu Bot Disallow	216	3.95
Yandex Bot Disallow	200	3.65
Sogou Bot Disallow	195	3.56
Alexa Bot Disallow	177	15.8
Microsoft BingBot Disallow	176	3.22
Yahoo Slurp Disallow	176	3.22
Soso Bot Disallow	160	2.92
GPTBot Disallow	125	11.16
Common Crawl Bot Disallow	125	11.16
AppleBot Disallow	116	2.12
Some Robots Disallow	112	2.05
Anthropic Claude Bot Disallow	112	10
Google Bard Disallow	112	10
Facebook Bot Disallow	110	9.82
Apple AI Bot Disallow	108	9.64
Perplexity	108	9.64
Cohere AI Disallow	103	9.2
Google AdsBot Mobile Apps Disallow	102	1.86
Google Media Partners Disallow	93 ^*	1.7
Majestic Bot Disallow	69	1.26
Semrush Bot Disallow	55	1
Microsoft MSNBot Disallow	23	0.42
Exalead Bot Disallow	15	0.27
ByteDance Bytespider Disallow	12	1.07
Google Image Bot Disallow	10	0.18
Webz Omgili Disallow	8	0.71
360 Bot Disallow	4	0.07
BuiltWithBot Disallow	4	0.07
Diffbot Disallow	2	0.18
Google News Bot Disallow	2	0.04
Google Video Bot Disallow	2	0.04
YOU Disallow	1	0.09
DuckDuckGo Bot Disallow	1	0.02
Google Favicon Disallow	1	0.02
Google API Bot Disallow	1	0.02