Robots.txt Usage Distribution in Japan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	172,564	20.68
Google AdsBot Disallow	171,684	20.57
Google AdsBot Mobile Disallow	145,801	17.47
Ahrefs Bot Disallow	40,912	4.9
Robots Disallow	35,850	4.3
Baidu Bot Disallow	29,430	3.53
Majestic Bot Disallow	22,880	2.74
Semrush Bot Disallow	22,726	2.72
Microsoft BingBot Disallow	22,162	2.66
Sogou Bot Disallow	21,214	2.54
Yandex Bot Disallow	18,979	2.27
Alexa Bot Disallow	17,622	2.11
Yahoo Slurp Disallow	16,498	1.98
Soso Bot Disallow	15,926	1.91
Some Robots Disallow	15,641	1.87
GPTBot Disallow	8,497	1.02
AppleBot Disallow	5,921	0.71
Google Media Partners Disallow	5,913 ^*	0.71
Common Crawl Bot Disallow	5,585	0.67
Anthropic Claude Bot Disallow	5,202	0.62
Google Bard Disallow	4,575	0.55
Apple AI Bot Disallow	4,277	0.51
Facebook Bot Disallow	4,000	0.48
Perplexity	3,195	0.38
Cohere AI Disallow	2,999	0.36
Microsoft MSNBot Disallow	2,781	0.33
Google AdsBot Mobile Apps Disallow	2,639	0.32
ByteDance Bytespider Disallow	1,877	0.22
Exalead Bot Disallow	1,603	0.19
Webz Omgili Disallow	1,521	0.18
Google Image Bot Disallow	1,497	0.18
360 Bot Disallow	981	0.12
Youdao Bot Disallow	383	0.05
Diffbot Disallow	173	0.02
Google News Bot Disallow	163	0.02
DuckDuckGo Bot Disallow	158	0.02
Google Video Bot Disallow	145	0.02
Pipl Disallow	114	0.01
Google Favicon Disallow	113	0.01
Google API Bot Disallow	113	0.01
YOU Disallow	110	0.01
Google DuplexWeb Disallow	106	0.01
BuiltWithBot Disallow	94	0.01
Google FeedFetcher Disallow	3	0