Robots.txt Usage Distribution in Switzerland

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	111,016	12.31
Google AdsBot Disallow	107,964	11.97
Google AdsBot Mobile Disallow	94,483	10.48
Robots Disallow	41,743	4.63
Baidu Bot Disallow	30,409	3.37
Anthropic Claude Bot Disallow	28,153	3.12
GPTBot Disallow	27,963	3.1
Common Crawl Bot Disallow	27,586	3.06
Google Bard Disallow	26,078	2.89
Facebook Bot Disallow	25,681	2.85
AppleBot Disallow	25,544	2.83
Cohere AI Disallow	25,065	2.78
Apple AI Bot Disallow	24,960	2.77
Yandex Bot Disallow	24,904	2.76
Perplexity	24,861	2.76
Sogou Bot Disallow	22,945	2.54
Ahrefs Bot Disallow	22,637	2.51
Microsoft BingBot Disallow	21,551	2.39
Yahoo Slurp Disallow	21,298	2.36
Alexa Bot Disallow	19,834	2.2
Soso Bot Disallow	18,297	2.03
Google AdsBot Mobile Apps Disallow	17,922	1.99
Some Robots Disallow	13,418	1.49
ByteDance Bytespider Disallow	12,486	1.38
Majestic Bot Disallow	11,148	1.24
Semrush Bot Disallow	10,334	1.15
Webz Omgili Disallow	10,224	1.13
Diffbot Disallow	9,390	1.04
YOU Disallow	9,063	1
Google Media Partners Disallow	4,652 ^*	0.52
Google Image Bot Disallow	4,575	0.51
Microsoft MSNBot Disallow	4,228	0.47
DuckDuckGo Bot Disallow	3,347	0.37
Google News Bot Disallow	3,186	0.35
Google Video Bot Disallow	3,156	0.35
Exalead Bot Disallow	2,450	0.27
Google Favicon Disallow	2,273	0.25
Google API Bot Disallow	2,177	0.24
Google DuplexWeb Disallow	2,171	0.24
360 Bot Disallow	928	0.1
Youdao Bot Disallow	708	0.08
Pipl Disallow	683	0.08
BuiltWithBot Disallow	391	0.04
Google FeedFetcher Disallow	53	0.01
Google Read Aloud Disallow	3	0
Google Web Light Disallow	2	0