Robots.txt Usage Distribution in Latvia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	4,432	13.42
Google AdsBot Disallow	4,189	12.68
Google AdsBot Mobile Disallow	2,897	8.77
Ahrefs Bot Disallow	1,998	6.05
Robots Disallow	1,890	5.72
Yandex Bot Disallow	1,677	5.08
Baidu Bot Disallow	1,662	5.03
Alexa Bot Disallow	1,211	3.67
Microsoft BingBot Disallow	1,120	3.39
Sogou Bot Disallow	1,097	3.32
Soso Bot Disallow	1,087	3.29
Yahoo Slurp Disallow	1,081	3.27
Some Robots Disallow	1,026	3.11
Majestic Bot Disallow	932	2.82
Semrush Bot Disallow	773	2.34
Common Crawl Bot Disallow	622	1.88
GPTBot Disallow	522	1.58
Anthropic Claude Bot Disallow	466	1.41
Google Bard Disallow	447	1.35
Facebook Bot Disallow	436	1.32
AppleBot Disallow	427	1.29
Google AdsBot Mobile Apps Disallow	420	1.27
Apple AI Bot Disallow	391	1.18
Cohere AI Disallow	382	1.16
Perplexity	366	1.11
Google Media Partners Disallow	258 ^*	0.78
Exalead Bot Disallow	254	0.77
ByteDance Bytespider Disallow	166	0.5
Microsoft MSNBot Disallow	140	0.42
360 Bot Disallow	121	0.37
Google Image Bot Disallow	113	0.34
BuiltWithBot Disallow	101	0.31
Webz Omgili Disallow	65	0.2
Google Video Bot Disallow	49	0.15
Google News Bot Disallow	47	0.14
DuckDuckGo Bot Disallow	44	0.13
Google Favicon Disallow	34	0.1
Google API Bot Disallow	32	0.1
Google DuplexWeb Disallow	32	0.1
Diffbot Disallow	17	0.05
Pipl Disallow	7	0.02
YOU Disallow	3	0.01
Youdao Bot Disallow	2	0.01
Google Read Aloud Disallow	1	0