Robots.txt Usage Distribution in Isle of Man

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	1,910	11.68
Baidu Bot Disallow	1,837	11.23
Yandex Bot Disallow	1,675	10.24
Some Robots Disallow	1,234	7.54
Anthropic Claude Bot Disallow	1,101	38.99
Microsoft BingBot Disallow	847	5.18
Yahoo Slurp Disallow	782	4.78
Sogou Bot Disallow	779	4.76
Ahrefs Bot Disallow	776	4.74
Soso Bot Disallow	775	4.74
Alexa Bot Disallow	769	27.23
Google AdsBot Disallow	746	4.56
Robots Disallow	593	3.63
Semrush Bot Disallow	549	3.36
Majestic Bot Disallow	543	3.32
Google AdsBot Mobile Disallow	165	1.01
GPTBot Disallow	148	5.24
Common Crawl Bot Disallow	132	4.67
Google Bard Disallow	127	4.5
Facebook Bot Disallow	123	4.36
AppleBot Disallow	119	0.73
Apple AI Bot Disallow	112	3.97
Cohere AI Disallow	78	2.76
Google AdsBot Mobile Apps Disallow	77	0.47
Perplexity	75	2.66
Webz Omgili Disallow	58	2.05
Microsoft MSNBot Disallow	30	0.18
ByteDance Bytespider Disallow	29	1.03
Exalead Bot Disallow	22	0.13
Google Image Bot Disallow	18	0.11
Google Media Partners Disallow	16 ^*	0.1
360 Bot Disallow	15	0.09
DuckDuckGo Bot Disallow	13	0.08
BuiltWithBot Disallow	10	0.06
Google News Bot Disallow	10	0.06
Google Video Bot Disallow	10	0.06
Google Favicon Disallow	10	0.06
Google API Bot Disallow	10	0.06
Google DuplexWeb Disallow	10	0.06
Diffbot Disallow	9	0.32
Pipl Disallow	7	0.04
YOU Disallow	6	0.21
Youdao Bot Disallow	2	0.01