Robots.txt Usage Distribution in Palestine

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google Media Partners Disallow	113 ^*	12.34
Robots Disallow	113	12.34
Googlebot Disallow	89	9.72
Ahrefs Bot Disallow	62	6.77
Some Robots Disallow	45	4.91
GPTBot Disallow	38	13.87
Common Crawl Bot Disallow	32	11.68
Facebook Bot Disallow	31	11.31
Google AdsBot Mobile Apps Disallow	31	3.38
Yandex Bot Disallow	30	3.28
AppleBot Disallow	29	3.17
Anthropic Claude Bot Disallow	29	10.58
Apple AI Bot Disallow	29	10.58
Yahoo Slurp Disallow	28	3.06
Google Bard Disallow	28	10.22
Cohere AI Disallow	28	10.22
Perplexity	28	10.22
Microsoft BingBot Disallow	27	2.95
Sogou Bot Disallow	26	2.84
Alexa Bot Disallow	25	9.12
Soso Bot Disallow	24	2.62
Majestic Bot Disallow	10	1.09
Semrush Bot Disallow	10	1.09
Microsoft MSNBot Disallow	5	0.55
ByteDance Bytespider Disallow	2	0.73
Webz Omgili Disallow	2	0.73
Exalead Bot Disallow	1	0.11
Diffbot Disallow	1	0.36