Robots.txt Nutzungsverteilung in Saudi-Arabien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Ahrefs Bot Disallow	3.384	9,72
Sogou Bot Disallow	3.151	9,05
Common Crawl Bot Disallow	3.108	8,93
Soso Bot Disallow	3.069	8,82
Majestic Bot Disallow	2.958	8,5
Exalead Bot Disallow	2.874	8,26
360 Bot Disallow	2.865	8,23
BuiltWithBot Disallow	2.861	8,22
Google AdsBot Disallow	1.729	4,97
Robots Disallow	1.615	4,64
Googlebot Disallow	1.576	4,53
Google AdsBot Mobile Disallow	1.298	3,73
Baidu Bot Disallow	393	1,13
Some Robots Disallow	342	0,98
Yandex Bot Disallow	287	0,82
GPTBot Disallow	272	0,78
Google Bard Disallow	249	0,72
Anthropic Claude Bot Disallow	247	0,71
AppleBot Disallow	246	0,71
Microsoft BingBot Disallow	243	0,7
Alexa Bot Disallow	242	0,7
Yahoo Slurp Disallow	239	0,69
Facebook Bot Disallow	239	0,69
Perplexity	235	0,68
Apple AI Bot Disallow	233	0,67
Cohere AI Disallow	232	0,67
Google AdsBot Mobile Apps Disallow	231	0,66
Semrush Bot Disallow	136	0,39
Google Media Partners Disallow	80 ^*	0,23
Microsoft MSNBot Disallow	32	0,09
Google Image Bot Disallow	32	0,09
ByteDance Bytespider Disallow	22	0,06
Google News Bot Disallow	14	0,04
DuckDuckGo Bot Disallow	13	0,04
Google Video Bot Disallow	12	0,03
Webz Omgili Disallow	11	0,03
Diffbot Disallow	7	0,02
Google Favicon Disallow	7	0,02
Google API Bot Disallow	7	0,02
Google DuplexWeb Disallow	7	0,02
YOU Disallow	4	0,01
Youdao Bot Disallow	4	0,01
Pipl Disallow	1	0