Robots.txt Usage Distribution in Bosnia and Herzegovina

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	275	9.3
Robots Disallow	268	9.06
Google AdsBot Disallow	229	7.74
Ahrefs Bot Disallow	159	5.38
Google AdsBot Mobile Disallow	158	5.34
Sogou Bot Disallow	129	4.36
GPTBot Disallow	117	3.96
Soso Bot Disallow	112	3.79
Baidu Bot Disallow	107	3.62
Common Crawl Bot Disallow	105	3.55
Yandex Bot Disallow	89	3.01
Majestic Bot Disallow	87	2.94
Alexa Bot Disallow	82	2.77
Yahoo Slurp Disallow	81	2.74
Semrush Bot Disallow	78	2.64
Microsoft BingBot Disallow	75	2.54
Some Robots Disallow	64	2.16
Google Bard Disallow	64	2.16
Anthropic Claude Bot Disallow	60	2.03
AppleBot Disallow	58	1.96
ByteDance Bytespider Disallow	58	1.96
Facebook Bot Disallow	57	1.93
Exalead Bot Disallow	56	1.89
Perplexity	55	1.86
360 Bot Disallow	49	1.66
Cohere AI Disallow	47	1.59
Apple AI Bot Disallow	46	1.56
Google AdsBot Mobile Apps Disallow	45	1.52
BuiltWithBot Disallow	45	1.52
Microsoft MSNBot Disallow	19	0.64
Google Media Partners Disallow	16 ^*	0.54
Google Image Bot Disallow	16	0.54
Webz Omgili Disallow	13	0.44
Diffbot Disallow	12	0.41
YOU Disallow	11	0.37
Google News Bot Disallow	2	0.07
Google Video Bot Disallow	2	0.07
DuckDuckGo Bot Disallow	2	0.07
Google Favicon Disallow	2	0.07
Google API Bot Disallow	2	0.07
Google DuplexWeb Disallow	2	0.07
Youdao Bot Disallow	1	0.03
Pipl Disallow	1	0.03
Google FeedFetcher Disallow	1	0.03