Robots.txt Nutzungsverteilung in Singapur

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	24.753	20,61
Google AdsBot Mobile Disallow	17.882	14,89
Googlebot Disallow	15.646	13,02
Ahrefs Bot Disallow	6.166	5,13
Robots Disallow	4.152	3,46
GPTBot Disallow	3.807	3,17
Common Crawl Bot Disallow	3.735	3,11
AppleBot Disallow	3.715	3,09
Anthropic Claude Bot Disallow	3.715	3,09
Google Bard Disallow	3.676	3,06
Facebook Bot Disallow	3.626	3,02
Apple AI Bot Disallow	3.610	3,01
Perplexity	3.564	2,97
Cohere AI Disallow	3.540	2,95
Google AdsBot Mobile Apps Disallow	3.519	2,93
Baidu Bot Disallow	2.030	1,69
Some Robots Disallow	1.624	1,35
Microsoft BingBot Disallow	1.385	1,15
Sogou Bot Disallow	1.327	1,1
Yandex Bot Disallow	1.319	1,1
Majestic Bot Disallow	1.308	1,09
Alexa Bot Disallow	1.295	1,08
Yahoo Slurp Disallow	1.073	0,89
Soso Bot Disallow	925	0,77
Semrush Bot Disallow	772	0,64
Google Media Partners Disallow	498 ^*	0,41
Exalead Bot Disallow	325	0,27
Google Image Bot Disallow	241	0,2
Microsoft MSNBot Disallow	201	0,17
ByteDance Bytespider Disallow	148	0,12
Webz Omgili Disallow	145	0,12
Diffbot Disallow	66	0,05
DuckDuckGo Bot Disallow	50	0,04
Google News Bot Disallow	44	0,04
Google Video Bot Disallow	42	0,03
360 Bot Disallow	40	0,03
Google Favicon Disallow	39	0,03
YOU Disallow	29	0,02
Youdao Bot Disallow	22	0,02
Google API Bot Disallow	21	0,02
Google DuplexWeb Disallow	20	0,02
Pipl Disallow	19	0,02
BuiltWithBot Disallow	15	0,01