Robots.txt Usage Distribution in Croatia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	3,347	13.98
Googlebot Disallow	3,180	13.28
Google AdsBot Mobile Disallow	2,423	10.12
Robots Disallow	1,955	8.17
Ahrefs Bot Disallow	1,328	5.55
Baidu Bot Disallow	712	2.97
Yandex Bot Disallow	644	2.69
Common Crawl Bot Disallow	602	2.51
GPTBot Disallow	588	2.46
Alexa Bot Disallow	569	2.38
Microsoft BingBot Disallow	555	2.32
Yahoo Slurp Disallow	530	2.21
Majestic Bot Disallow	510	2.13
Sogou Bot Disallow	504	2.1
Anthropic Claude Bot Disallow	500	2.09
Google Bard Disallow	490	2.05
AppleBot Disallow	480	2
Soso Bot Disallow	476	1.99
Facebook Bot Disallow	473	1.98
Semrush Bot Disallow	454	1.9
Google AdsBot Mobile Apps Disallow	443	1.85
Apple AI Bot Disallow	436	1.82
Cohere AI Disallow	433	1.81
Perplexity	429	1.79
Some Robots Disallow	374	1.56
Google Image Bot Disallow	301	1.26
Exalead Bot Disallow	180	0.75
Microsoft MSNBot Disallow	180	0.75
Google Media Partners Disallow	145 ^*	0.61
Google Video Bot Disallow	100	0.42
Google News Bot Disallow	99	0.41
ByteDance Bytespider Disallow	94	0.39
360 Bot Disallow	93	0.39
BuiltWithBot Disallow	77	0.32
Webz Omgili Disallow	69	0.29
DuckDuckGo Bot Disallow	42	0.18
Google API Bot Disallow	26	0.11
Google Favicon Disallow	21	0.09
Google DuplexWeb Disallow	21	0.09
YOU Disallow	20	0.08
Diffbot Disallow	19	0.08
Google FeedFetcher Disallow	7	0.03
Pipl Disallow	5	0.02
Google Read Aloud Disallow	5	0.02
Youdao Bot Disallow	4	0.02