Robots.txt Nutzungsverteilung In Libyen

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google Media Partners Disallow	48 ^*	25,13
Robots Disallow	25	13,09
Common Crawl Bot Disallow	10	13,51
Ahrefs Bot Disallow	9	4,71
GPTBot Disallow	9	12,16
AppleBot Disallow	9	4,71
Anthropic Claude Bot Disallow	9	12,16
Google Bard Disallow	9	12,16
Facebook Bot Disallow	9	12,16
Google AdsBot Mobile Apps Disallow	9	4,71
Apple AI Bot Disallow	9	12,16
Cohere AI Disallow	9	12,16
Perplexity	9	12,16
Yandex Bot Disallow	3	1,57
Semrush Bot Disallow	3	1,57
Sogou Bot Disallow	2	1,05
Soso Bot Disallow	2	1,05
Majestic Bot Disallow	2	1,05
Microsoft BingBot Disallow	1	0,52
Alexa Bot Disallow	1	1,35
Yahoo Slurp Disallow	1	0,52
Exalead Bot Disallow	1	0,52
360 Bot Disallow	1	0,52
BuiltWithBot Disallow	1	0,52