Robots.txt Usage Distribution in Brazil

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	151,031	14.47
Googlebot Disallow	143,501	13.75
Google AdsBot Mobile Disallow	120,094	11.51
Robots Disallow	98,903	9.48
Ahrefs Bot Disallow	55,577	5.32
Baidu Bot Disallow	40,001	3.83
Anthropic Claude Bot Disallow	38,315	3.67
Majestic Bot Disallow	36,315	3.48
Yandex Bot Disallow	35,185	3.37
Yahoo Slurp Disallow	34,067	3.26
Exalead Bot Disallow	27,505	2.64
GPTBot Disallow	25,015	2.4
Common Crawl Bot Disallow	24,843	2.38
Perplexity	22,342	2.14
Cohere AI Disallow	22,298	2.14
Some Robots Disallow	21,865	2.09
ByteDance Bytespider Disallow	20,799	1.99
Webz Omgili Disallow	20,163	1.93
Diffbot Disallow	19,771	1.89
Sogou Bot Disallow	12,741	1.22
Microsoft BingBot Disallow	10,566	1.01
Alexa Bot Disallow	9,877	0.95
Semrush Bot Disallow	9,723	0.93
Soso Bot Disallow	8,236	0.79
Google Image Bot Disallow	6,522	0.62
Google Media Partners Disallow	4,179 ^*	0.4
AppleBot Disallow	4,012	0.38
Google Bard Disallow	3,353	0.32
Facebook Bot Disallow	3,062	0.29
Apple AI Bot Disallow	2,871	0.28
Google AdsBot Mobile Apps Disallow	2,610	0.25
Microsoft MSNBot Disallow	1,980	0.19
360 Bot Disallow	1,624	0.16
DuckDuckGo Bot Disallow	1,124	0.11
BuiltWithBot Disallow	1,060	0.1
Google FeedFetcher Disallow	610	0.06
YOU Disallow	518	0.05
Youdao Bot Disallow	443	0.04
Google News Bot Disallow	258	0.02
Google Video Bot Disallow	212	0.02
Google Favicon Disallow	166	0.02
Google API Bot Disallow	155	0.01
Google DuplexWeb Disallow	126	0.01
Pipl Disallow	80	0.01
Google Read Aloud Disallow	8	0
LLMS Text	3	0
Google Web Light Disallow	2	0