Robots.txt Usage Distribution in Ireland

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	27,253	10.21
Googlebot Disallow	21,835	8.18
Google AdsBot Mobile Disallow	20,134	7.54
Baidu Bot Disallow	10,839	4.06
GPTBot Disallow	10,789	4.04
Common Crawl Bot Disallow	10,732	4.02
Yandex Bot Disallow	10,674	4
AppleBot Disallow	10,627	3.98
Anthropic Claude Bot Disallow	10,544	3.95
Google Bard Disallow	10,530	3.94
Facebook Bot Disallow	10,485	3.93
Apple AI Bot Disallow	10,434	3.91
Perplexity	10,370	3.88
Microsoft BingBot Disallow	10,358	3.88
Cohere AI Disallow	10,354	3.88
Alexa Bot Disallow	10,316	3.86
Yahoo Slurp Disallow	10,275	3.85
Sogou Bot Disallow	10,212	3.82
Soso Bot Disallow	10,122	3.79
Google AdsBot Mobile Apps Disallow	9,369	3.51
Ahrefs Bot Disallow	7,816	2.93
Robots Disallow	2,525	0.95
Some Robots Disallow	1,300	0.49
ByteDance Bytespider Disallow	1,204	0.45
Webz Omgili Disallow	1,183	0.44
Majestic Bot Disallow	1,176	0.44
Diffbot Disallow	1,042	0.39
YOU Disallow	992	0.37
Semrush Bot Disallow	964	0.36
Google Media Partners Disallow	778 ^*	0.29
Microsoft MSNBot Disallow	374	0.14
Exalead Bot Disallow	348	0.13
Google Image Bot Disallow	305	0.11
360 Bot Disallow	160	0.06
BuiltWithBot Disallow	106	0.04
Google News Bot Disallow	103	0.04
Google Video Bot Disallow	96	0.04
Google Favicon Disallow	85	0.03
DuckDuckGo Bot Disallow	64	0.02
Google API Bot Disallow	46	0.02
Google DuplexWeb Disallow	46	0.02
Youdao Bot Disallow	17	0.01
Pipl Disallow	16	0.01
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	2	0