Robots.txt Usage Distribution in New Zealand

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	44,815	13.62
Google AdsBot Mobile Disallow	32,075	9.75
Googlebot Disallow	20,254	6.15
GPTBot Disallow	17,827	5.42
Common Crawl Bot Disallow	17,597	5.35
Anthropic Claude Bot Disallow	17,483	5.31
AppleBot Disallow	17,319	5.26
Google Bard Disallow	17,282	5.25
Facebook Bot Disallow	17,203	5.23
Apple AI Bot Disallow	17,115	5.2
Perplexity	17,007	5.17
Cohere AI Disallow	16,986	5.16
Google AdsBot Mobile Apps Disallow	16,925	5.14
Ahrefs Bot Disallow	14,058	4.27
Baidu Bot Disallow	8,364	2.54
Microsoft BingBot Disallow	6,022	1.83
Robots Disallow	5,519	1.68
Yandex Bot Disallow	4,099	1.25
Sogou Bot Disallow	3,568	1.08
Yahoo Slurp Disallow	3,379	1.03
Alexa Bot Disallow	2,507	0.76
Soso Bot Disallow	2,268	0.69
Majestic Bot Disallow	1,991	0.6
Some Robots Disallow	1,930	0.59
Semrush Bot Disallow	1,595	0.48
Google Media Partners Disallow	584 ^*	0.18
ByteDance Bytespider Disallow	549	0.17
Microsoft MSNBot Disallow	547	0.17
Exalead Bot Disallow	459	0.14
Google Image Bot Disallow	445	0.14
Webz Omgili Disallow	375	0.11
Diffbot Disallow	141	0.04
360 Bot Disallow	140	0.04
Google News Bot Disallow	118	0.04
Google Video Bot Disallow	113	0.03
BuiltWithBot Disallow	87	0.03
Google Favicon Disallow	80	0.02
YOU Disallow	75	0.02
DuckDuckGo Bot Disallow	62	0.02
Google API Bot Disallow	39	0.01
Google DuplexWeb Disallow	38	0.01
Youdao Bot Disallow	31	0.01
Pipl Disallow	23	0.01
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0