Robots.txt Usage Distribution in Qatar

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	1,091	16.63
Googlebot Disallow	849	12.94
Google AdsBot Mobile Disallow	648	9.88
Robots Disallow	432	6.59
Ahrefs Bot Disallow	408	6.22
Baidu Bot Disallow	278	4.24
Sogou Bot Disallow	236	3.6
Yandex Bot Disallow	218	3.32
Alexa Bot Disallow	196	2.99
Microsoft BingBot Disallow	193	2.94
Yahoo Slurp Disallow	193	2.94
Soso Bot Disallow	176	2.68
Some Robots Disallow	136	2.07
Common Crawl Bot Disallow	136	2.07
GPTBot Disallow	132	2.01
AppleBot Disallow	123	1.88
Anthropic Claude Bot Disallow	122	1.86
Google Bard Disallow	121	1.84
Facebook Bot Disallow	119	1.81
Apple AI Bot Disallow	117	1.78
Perplexity	117	1.78
Cohere AI Disallow	113	1.72
Google AdsBot Mobile Apps Disallow	109	1.66
Majestic Bot Disallow	74	1.13
Google Media Partners Disallow	73 ^*	1.11
Semrush Bot Disallow	55	0.84
Microsoft MSNBot Disallow	23	0.35
Exalead Bot Disallow	16	0.24
ByteDance Bytespider Disallow	15	0.23
Google Image Bot Disallow	10	0.15
Webz Omgili Disallow	9	0.14
360 Bot Disallow	5	0.08
BuiltWithBot Disallow	5	0.08
Diffbot Disallow	3	0.05
YOU Disallow	2	0.03
Google News Bot Disallow	2	0.03
Google Video Bot Disallow	2	0.03
DuckDuckGo Bot Disallow	1	0.02
Google Favicon Disallow	1	0.02
Google API Bot Disallow	1	0.02