Robots.txt Usage Distribution in Czech Republic

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	39,868	13.28
Baidu Bot Disallow	25,690	8.56
Yandex Bot Disallow	21,605	7.2
Microsoft BingBot Disallow	20,632	6.87
Sogou Bot Disallow	20,609	6.87
Alexa Bot Disallow	19,421	6.47
Yahoo Slurp Disallow	19,303	6.43
Soso Bot Disallow	18,311	6.1
Google AdsBot Disallow	18,141	6.04
Robots Disallow	16,035	5.34
Google AdsBot Mobile Disallow	15,798	5.26
Ahrefs Bot Disallow	7,916	2.64
Semrush Bot Disallow	6,293	2.1
Majestic Bot Disallow	6,090	2.03
GPTBot Disallow	4,158	1.39
Webz Omgili Disallow	4,024	1.34
Anthropic Claude Bot Disallow	3,607	1.2
Some Robots Disallow	3,571	1.19
Common Crawl Bot Disallow	3,505	1.17
AppleBot Disallow	2,641	0.88
Google Bard Disallow	2,278	0.76
Facebook Bot Disallow	2,235	0.74
ByteDance Bytespider Disallow	2,049	0.68
Apple AI Bot Disallow	2,034	0.68
Cohere AI Disallow	1,895	0.63
Perplexity	1,847	0.62
Microsoft MSNBot Disallow	1,582	0.53
Google AdsBot Mobile Apps Disallow	1,345	0.45
Google Media Partners Disallow	1,229 ^*	0.41
Exalead Bot Disallow	1,126	0.38
Diffbot Disallow	950	0.32
YOU Disallow	857	0.29
Google Image Bot Disallow	721	0.24
Google News Bot Disallow	400	0.13
Google Video Bot Disallow	398	0.13
Google Favicon Disallow	394	0.13
DuckDuckGo Bot Disallow	393	0.13
Google API Bot Disallow	368	0.12
Google DuplexWeb Disallow	366	0.12
360 Bot Disallow	273	0.09
BuiltWithBot Disallow	116	0.04
Pipl Disallow	26	0.01
Youdao Bot Disallow	19	0.01
Google FeedFetcher Disallow	3	0
Google Read Aloud Disallow	3	0
LLMS Text	2	0