로봇.txt 사용량 분포 상위 100만개 사이트에

다음을 사용하여 웹사이트에 배포 로봇.txt 기술

기술	웹사이트	%
Robots Disallow	141,874	14.19
Ahrefs Bot Disallow	62,871	6.29
Googlebot Disallow	61,902	6.19
Yandex Bot Disallow	46,961	4.7
Google AdsBot Disallow	42,924	4.29
Some Robots Disallow	39,834	3.98
Baidu Bot Disallow	37,242	3.72
Semrush Bot Disallow	31,553	3.16
Majestic Bot Disallow	31,245	3.12
GPTBot Disallow	28,961	2.9
Microsoft BingBot Disallow	26,704	2.67
Google Media Partners Disallow	24,605	2.46
Yahoo Slurp Disallow	21,935	2.19
Alexa Bot Disallow	21,420	2.14
Common Crawl Bot Disallow	21,345	2.13
Sogou Bot Disallow	17,024	1.7
Anthropic Claude Bot Disallow	16,186	1.62
Soso Bot Disallow	14,362	1.44
Google Bard Disallow	13,614	1.36
Microsoft MSNBot Disallow	13,203	1.32
AppleBot Disallow	13,144	1.31
Google AdsBot Mobile Disallow	12,076	1.21
Facebook Bot Disallow	11,075	1.11
Google Image Bot Disallow	10,882	1.09
ByteDance Bytespider Disallow	9,397	0.94
Exalead Bot Disallow	9,186	0.92
Perplexity	8,203	0.82
Google AdsBot Mobile Apps Disallow	7,907	0.79
Cohere AI Disallow	7,549	0.75
Webz Omgili Disallow	7,446	0.74
Apple AI Bot Disallow	7,441	0.74
Google News Bot Disallow	6,995	0.7
DuckDuckGo Bot Disallow	6,041	0.6
Google Video Bot Disallow	5,510	0.55
360 Bot Disallow	5,484	0.55
Google Favicon Disallow	4,858	0.49
Google API Bot Disallow	4,769	0.48
Google DuplexWeb Disallow	4,653	0.47
Diffbot Disallow	3,794	0.38
YOU Disallow	2,809	0.28
BuiltWithBot Disallow	2,493	0.25
Pipl Disallow	2,108	0.21
Youdao Bot Disallow	948	0.09
Google FeedFetcher Disallow	151	0.02
Google Read Aloud Disallow	52	0.01
Google Web Light Disallow	16	0
Google BIG Bench Disallow	4	0