Robots.txt Usage Distribution in France

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	196,776	14.93
Google AdsBot Disallow	193,088	14.65
Google AdsBot Mobile Disallow	147,143	11.16
Robots Disallow	79,424	6.02
Ahrefs Bot Disallow	70,302	5.33
Baidu Bot Disallow	53,412	4.05
Yandex Bot Disallow	36,453	2.77
Some Robots Disallow	31,976	2.43
Microsoft BingBot Disallow	30,893	2.34
Semrush Bot Disallow	30,267	2.3
Yahoo Slurp Disallow	29,377	2.23
GPTBot Disallow	28,665	13.18
Majestic Bot Disallow	28,372	2.15
Sogou Bot Disallow	27,326	2.07
Soso Bot Disallow	26,096	1.98
Common Crawl Bot Disallow	25,308	11.63
Alexa Bot Disallow	24,717	11.36
Anthropic Claude Bot Disallow	23,351	10.73
Google AdsBot Mobile Apps Disallow	22,422	1.7
Google Bard Disallow	21,266	9.78
Facebook Bot Disallow	20,552	9.45
AppleBot Disallow	20,434	1.55
Apple AI Bot Disallow	18,417	8.47
Perplexity	18,142	8.34
Cohere AI Disallow	16,675	7.67
Google Image Bot Disallow	13,074	0.99
Google Media Partners Disallow	10,565 ^*	0.8
Microsoft MSNBot Disallow	10,068	0.76
DuckDuckGo Bot Disallow	7,205	0.55
Youdao Bot Disallow	7,003	0.53
ByteDance Bytespider Disallow	6,586	3.03
Exalead Bot Disallow	6,348	0.48
Google News Bot Disallow	5,941	0.45
Google Video Bot Disallow	5,909	0.45
Webz Omgili Disallow	5,129	2.36
Google Favicon Disallow	4,745	0.36
Google API Bot Disallow	4,689	0.36
Google DuplexWeb Disallow	4,666	0.35
360 Bot Disallow	2,634	0.2
Diffbot Disallow	988	0.45
YOU Disallow	812	0.37
BuiltWithBot Disallow	629	0.05
Pipl Disallow	465	0.04
Google FeedFetcher Disallow	28	0
Google Read Aloud Disallow	13	0
Google Web Light Disallow	4	0