Robots.txt Usage Distribution in Slovakia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	5,719	6.54
Baidu Bot Disallow	4,783	5.47
Ahrefs Bot Disallow	4,665	5.34
Robots Disallow	4,523	5.18
Google AdsBot Disallow	4,253	4.87
Semrush Bot Disallow	3,978	4.55
Majestic Bot Disallow	3,922	4.49
GPTBot Disallow	3,527	4.04
Yandex Bot Disallow	3,381	3.87
Some Robots Disallow	3,288	3.76
Google AdsBot Mobile Disallow	3,188	3.65
Microsoft BingBot Disallow	3,060	3.5
Facebook Bot Disallow	2,979	3.41
Common Crawl Bot Disallow	2,954	3.38
Anthropic Claude Bot Disallow	2,754	3.15
AppleBot Disallow	2,495	2.86
Google Bard Disallow	2,299	2.63
Alexa Bot Disallow	2,261	2.59
Apple AI Bot Disallow	2,214	2.53
Cohere AI Disallow	2,165	2.48
Sogou Bot Disallow	2,163	2.48
Perplexity	2,150	2.46
ByteDance Bytespider Disallow	2,024	2.32
Yahoo Slurp Disallow	2,014	2.3
Webz Omgili Disallow	2,011	2.3
Soso Bot Disallow	1,794	2.05
Diffbot Disallow	1,625	1.86
YOU Disallow	1,594	1.82
Microsoft MSNBot Disallow	875	1
Exalead Bot Disallow	697	0.8
Google AdsBot Mobile Apps Disallow	623	0.71
Google Media Partners Disallow	486 ^*	0.56
Google Image Bot Disallow	238	0.27
360 Bot Disallow	145	0.17
Google News Bot Disallow	87	0.1
Google Video Bot Disallow	87	0.1
DuckDuckGo Bot Disallow	87	0.1
Google Favicon Disallow	80	0.09
Google API Bot Disallow	76	0.09
Google DuplexWeb Disallow	75	0.09
BuiltWithBot Disallow	35	0.04
Youdao Bot Disallow	7	0.01
Pipl Disallow	5	0.01
Google Read Aloud Disallow	1	0