Robots.txt Usage Distribution in Greenland

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	117	9.54
Googlebot Disallow	108	8.81
Google AdsBot Disallow	106	8.65
Google AdsBot Mobile Disallow	83	6.77
Baidu Bot Disallow	54	4.4
Ahrefs Bot Disallow	51	4.16
GPTBot Disallow	47	3.83
Common Crawl Bot Disallow	45	3.67
Anthropic Claude Bot Disallow	39	3.18
Google Bard Disallow	39	3.18
Facebook Bot Disallow	38	3.1
Some Robots Disallow	37	3.02
Google AdsBot Mobile Apps Disallow	37	3.02
AppleBot Disallow	35	2.85
Apple AI Bot Disallow	34	2.77
Cohere AI Disallow	33	2.69
Google Media Partners Disallow	32 ^*	2.61
Perplexity	32	2.61
Yandex Bot Disallow	31	2.53
Microsoft BingBot Disallow	30	2.45
Alexa Bot Disallow	29	2.37
Yahoo Slurp Disallow	28	2.28
Sogou Bot Disallow	24	1.96
Soso Bot Disallow	24	1.96
Majestic Bot Disallow	24	1.96
Semrush Bot Disallow	9	0.73
Microsoft MSNBot Disallow	8	0.65
Webz Omgili Disallow	8	0.65
ByteDance Bytespider Disallow	7	0.57
Google Image Bot Disallow	6	0.49
Google News Bot Disallow	4	0.33
Google Video Bot Disallow	4	0.33
DuckDuckGo Bot Disallow	4	0.33
Google Favicon Disallow	4	0.33
Google API Bot Disallow	4	0.33
Google DuplexWeb Disallow	4	0.33
Exalead Bot Disallow	3	0.24
Diffbot Disallow	2	0.16
360 Bot Disallow	1	0.08
BuiltWithBot Disallow	1	0.08