Robots.txt Nutzungsverteilung in Serbien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	1.856	11,17
Googlebot Disallow	1.831	11,02
Google AdsBot Disallow	1.244	7,49
Baidu Bot Disallow	1.030	6,2
Yandex Bot Disallow	1.011	6,08
Sogou Bot Disallow	964	5,8
Ahrefs Bot Disallow	900	5,42
Google AdsBot Mobile Disallow	828	4,98
Microsoft BingBot Disallow	766	4,61
Alexa Bot Disallow	747	30,06
Soso Bot Disallow	734	4,42
Yahoo Slurp Disallow	724	4,36
Some Robots Disallow	510	3,07
Majestic Bot Disallow	432	2,6
Common Crawl Bot Disallow	313	12,6
Google Media Partners Disallow	305 ^*	1,84
Semrush Bot Disallow	304	1,83
GPTBot Disallow	207	8,33
Exalead Bot Disallow	190	1,14
Anthropic Claude Bot Disallow	182	7,32
AppleBot Disallow	157	0,94
Google Bard Disallow	152	6,12
Facebook Bot Disallow	141	5,67
Apple AI Bot Disallow	127	5,11
Perplexity	126	5,07
Cohere AI Disallow	125	5,03
Google AdsBot Mobile Apps Disallow	115	0,69
Google Image Bot Disallow	76	0,46
360 Bot Disallow	72	0,43
Microsoft MSNBot Disallow	65	0,39
BuiltWithBot Disallow	58	0,35
ByteDance Bytespider Disallow	55	2,21
Webz Omgili Disallow	52	2,09
YOU Disallow	37	1,49
Diffbot Disallow	35	1,41
DuckDuckGo Bot Disallow	27	0,16
Google Video Bot Disallow	24	0,14
Google News Bot Disallow	23	0,14
Google API Bot Disallow	22	0,13
Google Favicon Disallow	21	0,13
Google DuplexWeb Disallow	21	0,13
Pipl Disallow	4	0,02
Youdao Bot Disallow	3	0,02