Robots.txt Nutzungsverteilung In Weißrussland

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Yandex Bot Disallow	6.414	34,11
Robots Disallow	2.660	14,14
Googlebot Disallow	2.459	13,08
Semrush Bot Disallow	712	3,79
Microsoft BingBot Disallow	561	2,98
Ahrefs Bot Disallow	546	2,9
Majestic Bot Disallow	502	2,67
Alexa Bot Disallow	483	18,17
Some Robots Disallow	470	2,5
GPTBot Disallow	375	14,11
Common Crawl Bot Disallow	368	13,84
Google Media Partners Disallow	339 ^*	1,8
Anthropic Claude Bot Disallow	310	11,66
Google Bard Disallow	305	11,47
Baidu Bot Disallow	293	1,56
Microsoft MSNBot Disallow	293	1,56
Yahoo Slurp Disallow	280	1,49
Webz Omgili Disallow	275	10,35
Sogou Bot Disallow	197	1,05
Soso Bot Disallow	195	1,04
Google AdsBot Disallow	135	0,72
Google AdsBot Mobile Disallow	107	0,57
Exalead Bot Disallow	102	0,54
Google Image Bot Disallow	83	0,44
AppleBot Disallow	45	0,24
Facebook Bot Disallow	37	1,39
Google AdsBot Mobile Apps Disallow	33	0,18
Apple AI Bot Disallow	31	1,17
360 Bot Disallow	28	0,15
Cohere AI Disallow	24	0,9
Perplexity	22	0,83
Google News Bot Disallow	17	0,09
DuckDuckGo Bot Disallow	17	0,09
ByteDance Bytespider Disallow	15	0,56
Google Video Bot Disallow	15	0,08
Google API Bot Disallow	13	0,07
Google Favicon Disallow	12	0,06
Google DuplexWeb Disallow	12	0,06
BuiltWithBot Disallow	10	0,05
Diffbot Disallow	4	0,15
YOU Disallow	2	0,08
Pipl Disallow	2	0,01
Google FeedFetcher Disallow	2	0,01
Youdao Bot Disallow	1	0,01