Robots.txt Nutzungsverteilung In Malawi

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	102	15,62
Robots Disallow	60	9,19
Google Media Partners Disallow	48 ^*	7,35
Common Crawl Bot Disallow	32	12,96
GPTBot Disallow	30	12,15
AppleBot Disallow	28	4,29
Anthropic Claude Bot Disallow	27	10,93
Google Bard Disallow	27	10,93
Facebook Bot Disallow	27	10,93
Google AdsBot Mobile Apps Disallow	27	4,13
Apple AI Bot Disallow	27	10,93
Cohere AI Disallow	26	10,53
Perplexity	26	10,53
Yandex Bot Disallow	20	3,06
Some Robots Disallow	18	2,76
Sogou Bot Disallow	16	2,45
Microsoft BingBot Disallow	15	2,3
Ahrefs Bot Disallow	15	2,3
Alexa Bot Disallow	15	6,07
Yahoo Slurp Disallow	15	2,3
Soso Bot Disallow	15	2,3
Majestic Bot Disallow	9	1,38
Semrush Bot Disallow	9	1,38
Exalead Bot Disallow	3	0,46
ByteDance Bytespider Disallow	2	0,81
Webz Omgili Disallow	2	0,81
Diffbot Disallow	2	0,81
Microsoft MSNBot Disallow	1	0,15
Google Image Bot Disallow	1	0,15
360 Bot Disallow	1	0,15
Google News Bot Disallow	1	0,15
Google Video Bot Disallow	1	0,15
DuckDuckGo Bot Disallow	1	0,15
Google Favicon Disallow	1	0,15
Google API Bot Disallow	1	0,15
Google DuplexWeb Disallow	1	0,15
YOU Disallow	1	0,4