Robots.txt Usage Distribution in Kuwait

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	1,114	19.28
Google AdsBot Mobile Disallow	608	10.52
Ahrefs Bot Disallow	559	9.67
Googlebot Disallow	540	9.35
Robots Disallow	285	4.93
Common Crawl Bot Disallow	214	13.41
GPTBot Disallow	209	13.1
Google AdsBot Mobile Apps Disallow	193	3.34
Google Bard Disallow	189	11.84
AppleBot Disallow	187	3.24
Anthropic Claude Bot Disallow	186	11.65
Facebook Bot Disallow	185	11.59
Perplexity	179	11.22
Apple AI Bot Disallow	178	11.15
Cohere AI Disallow	178	11.15
Baidu Bot Disallow	110	1.9
Some Robots Disallow	78	1.35
Yandex Bot Disallow	74	1.28
Yahoo Slurp Disallow	62	1.07
Microsoft BingBot Disallow	58	1
Alexa Bot Disallow	51	3.2
Sogou Bot Disallow	48	0.83
Soso Bot Disallow	48	0.83
Majestic Bot Disallow	47	0.81
Semrush Bot Disallow	37	0.64
Google Media Partners Disallow	32 ^*	0.55
Google Image Bot Disallow	23	0.4
Microsoft MSNBot Disallow	22	0.38
Exalead Bot Disallow	16	0.28
360 Bot Disallow	10	0.17
BuiltWithBot Disallow	10	0.17
Google News Bot Disallow	10	0.17
Google Video Bot Disallow	10	0.17
Webz Omgili Disallow	7	0.44
DuckDuckGo Bot Disallow	7	0.12
ByteDance Bytespider Disallow	6	0.38
Diffbot Disallow	2	0.13
Google Favicon Disallow	2	0.03
Google API Bot Disallow	2	0.03
Google DuplexWeb Disallow	2	0.03