Robots.txt Nutzungsverteilung In Jemen

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	35	14,96
Yandex Bot Disallow	25	10,68
Microsoft BingBot Disallow	21	8,97
Alexa Bot Disallow	21	34,43
Yahoo Slurp Disallow	20	8,55
Soso Bot Disallow	19	8,12
Ahrefs Bot Disallow	14	5,98
Common Crawl Bot Disallow	7	11,48
Majestic Bot Disallow	7	2,99
GPTBot Disallow	6	9,84
Semrush Bot Disallow	6	2,56
Google Bard Disallow	5	8,2
Google AdsBot Mobile Apps Disallow	5	2,14
AppleBot Disallow	4	1,71
Anthropic Claude Bot Disallow	4	6,56
Facebook Bot Disallow	4	6,56
Apple AI Bot Disallow	4	6,56
Cohere AI Disallow	4	6,56
Perplexity	4	6,56
Microsoft MSNBot Disallow	4	1,71
Google Image Bot Disallow	4	1,71
ByteDance Bytespider Disallow	2	3,28
Exalead Bot Disallow	1	0,43
360 Bot Disallow	1	0,43
BuiltWithBot Disallow	1	0,43
Google News Bot Disallow	1	0,43
Google Video Bot Disallow	1	0,43
DuckDuckGo Bot Disallow	1	0,43
Google Favicon Disallow	1	0,43
Google API Bot Disallow	1	0,43
Google DuplexWeb Disallow	1	0,43