Robots.txt Usage Distribution in Ukraine

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	21,305	25.42
Robots Disallow	14,116	16.84
Googlebot Disallow	9,825	11.72
Google AdsBot Disallow	3,706	4.42
Google AdsBot Mobile Disallow	3,108	3.71
Ahrefs Bot Disallow	2,753	3.28
Microsoft BingBot Disallow	2,478	2.96
Alexa Bot Disallow	2,473	29.61
Baidu Bot Disallow	2,455	2.93
Sogou Bot Disallow	2,255	2.69
Majestic Bot Disallow	2,243	2.68
Yahoo Slurp Disallow	2,168	2.59
Google Media Partners Disallow	1,950 ^*	2.33
Soso Bot Disallow	1,760	2.1
Semrush Bot Disallow	1,648	1.97
Some Robots Disallow	1,545	1.84
GPTBot Disallow	1,023	12.25
Common Crawl Bot Disallow	886	10.61
Microsoft MSNBot Disallow	704	0.84
Exalead Bot Disallow	566	0.68
Anthropic Claude Bot Disallow	494	5.91
Google Image Bot Disallow	493	0.59
Google Bard Disallow	441	5.28
360 Bot Disallow	414	0.49
AppleBot Disallow	354	0.42
Facebook Bot Disallow	307	3.68
Apple AI Bot Disallow	257	3.08
ByteDance Bytespider Disallow	254	3.04
Perplexity	252	3.02
Cohere AI Disallow	243	2.91
Youdao Bot Disallow	219	0.26
Google AdsBot Mobile Apps Disallow	198	0.24
Webz Omgili Disallow	175	2.1
YOU Disallow	166	1.99
BuiltWithBot Disallow	133	0.16
Diffbot Disallow	87	1.04
DuckDuckGo Bot Disallow	87	0.1
Google News Bot Disallow	74	0.09
Google Video Bot Disallow	57	0.07
Google API Bot Disallow	43	0.05
Google Favicon Disallow	41	0.05
Google DuplexWeb Disallow	39	0.05
Pipl Disallow	15	0.02
Google FeedFetcher Disallow	3	0
LLMS Text	3	0
Google Read Aloud Disallow	1	0