Robots.txt Usage Distribution in China

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	66,632	10.98
Ahrefs Bot Disallow	53,827	8.87
Semrush Bot Disallow	42,532	7.01
Majestic Bot Disallow	38,788	6.39
Some Robots Disallow	36,694	6.05
Robots Disallow	34,985	5.77
Google AdsBot Disallow	33,693	5.55
Microsoft BingBot Disallow	33,567	5.53
Baidu Bot Disallow	32,165	5.3
Yahoo Slurp Disallow	26,317	4.34
Alexa Bot Disallow	25,479	35.2
Yandex Bot Disallow	24,502	4.04
Google AdsBot Mobile Disallow	23,010	3.79
Microsoft MSNBot Disallow	22,007	3.63
Google Image Bot Disallow	16,169	2.67
Sogou Bot Disallow	14,851	2.45
AppleBot Disallow	14,582	2.4
Soso Bot Disallow	14,059	2.32
GPTBot Disallow	8,369	11.56
Anthropic Claude Bot Disallow	7,033	9.72
360 Bot Disallow	4,049	0.67
ByteDance Bytespider Disallow	4,035	5.57
Common Crawl Bot Disallow	3,784	5.23
Exalead Bot Disallow	3,684	0.61
Youdao Bot Disallow	3,346	0.55
Google Bard Disallow	2,729	3.77
Facebook Bot Disallow	2,721	3.76
Google AdsBot Mobile Apps Disallow	2,386	0.39
Cohere AI Disallow	2,157	2.98
Apple AI Bot Disallow	2,137	2.95
Perplexity	2,068	2.86
Google Media Partners Disallow	1,018 ^*	0.17
DuckDuckGo Bot Disallow	395	0.07
Webz Omgili Disallow	389	0.54
Google News Bot Disallow	351	0.06
Google Video Bot Disallow	345	0.06
BuiltWithBot Disallow	329	0.05
Google Favicon Disallow	297	0.05
Google API Bot Disallow	292	0.05
Google DuplexWeb Disallow	291	0.05
Google FeedFetcher Disallow	233	0.04
Diffbot Disallow	231	0.32
YOU Disallow	41	0.06
Pipl Disallow	18	0