Robots.txt Usage Distribution in Kuwait

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	1,074	18.71
Google AdsBot Mobile Disallow	576	10.04
Ahrefs Bot Disallow	560	9.76
Googlebot Disallow	523	9.11
Robots Disallow	248	4.32
Common Crawl Bot Disallow	208	3.62
GPTBot Disallow	196	3.42
AppleBot Disallow	195	3.4
Google Bard Disallow	195	3.4
Anthropic Claude Bot Disallow	191	3.33
Facebook Bot Disallow	191	3.33
Perplexity	189	3.29
Apple AI Bot Disallow	187	3.26
Cohere AI Disallow	185	3.22
Google AdsBot Mobile Apps Disallow	183	3.19
Baidu Bot Disallow	106	1.85
Yandex Bot Disallow	87	1.52
Some Robots Disallow	81	1.41
Sogou Bot Disallow	71	1.24
Microsoft BingBot Disallow	71	1.24
Yahoo Slurp Disallow	68	1.18
Soso Bot Disallow	66	1.15
Alexa Bot Disallow	65	1.13
Majestic Bot Disallow	52	0.91
Semrush Bot Disallow	38	0.66
Google Media Partners Disallow	32 ^*	0.56
Exalead Bot Disallow	20	0.35
Microsoft MSNBot Disallow	14	0.24
360 Bot Disallow	14	0.24
Google Image Bot Disallow	14	0.24
BuiltWithBot Disallow	13	0.23
ByteDance Bytespider Disallow	7	0.12
Webz Omgili Disallow	7	0.12
DuckDuckGo Bot Disallow	5	0.09
Diffbot Disallow	3	0.05
Google News Bot Disallow	2	0.03
Google Video Bot Disallow	2	0.03