Robots.txt Usage Distribution in Palestine

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google Media Partners Disallow	113 ^*	12.34
Robots Disallow	113	12.34
Googlebot Disallow	89	9.72
Ahrefs Bot Disallow	62	6.77
Some Robots Disallow	45	4.91
GPTBot Disallow	38	4.15
Common Crawl Bot Disallow	32	3.49
Facebook Bot Disallow	31	3.38
Google AdsBot Mobile Apps Disallow	31	3.38
Yandex Bot Disallow	30	3.28
AppleBot Disallow	29	3.17
Anthropic Claude Bot Disallow	29	3.17
Apple AI Bot Disallow	29	3.17
Yahoo Slurp Disallow	28	3.06
Google Bard Disallow	28	3.06
Cohere AI Disallow	28	3.06
Perplexity	28	3.06
Microsoft BingBot Disallow	27	2.95
Sogou Bot Disallow	26	2.84
Alexa Bot Disallow	25	2.73
Soso Bot Disallow	24	2.62
Majestic Bot Disallow	10	1.09
Semrush Bot Disallow	10	1.09
Microsoft MSNBot Disallow	5	0.55
ByteDance Bytespider Disallow	2	0.22
Webz Omgili Disallow	2	0.22
Exalead Bot Disallow	1	0.11
Diffbot Disallow	1	0.11