로봇.txt 사용 분포 상위 100k 사이트

웹사이트를 위한 배포 로봇.txt 기술

기술	웹사이트	%
Robots Disallow	29,575	29.58
Googlebot Disallow	9,749	9.75
Some Robots Disallow	9,253	9.25
Ahrefs Bot Disallow	8,931	8.93
GPTBot Disallow	8,235	8.24
Yandex Bot Disallow	6,458	6.46
Semrush Bot Disallow	6,119	6.12
Common Crawl Bot Disallow	5,885	5.88
AppleBot Disallow	5,811	5.81
Google AdsBot Disallow	5,699	5.7
Majestic Bot Disallow	5,302	5.3
Anthropic Claude Bot Disallow	5,036	5.04
Baidu Bot Disallow	4,781	4.78
Microsoft BingBot Disallow	4,315	4.32
Google Bard Disallow	4,201	4.2
ByteDance Bytespider Disallow	3,551	3.55
Facebook Bot Disallow	3,405	3.4
Perplexity	3,071	3.07
Alexa Bot Disallow	2,930	2.93
Webz Omgili Disallow	2,929	2.93
Cohere AI Disallow	2,729	2.73
Apple AI Bot Disallow	2,703	2.7
Yahoo Slurp Disallow	2,680	2.68
Google Media Partners Disallow	2,606	2.61
Microsoft MSNBot Disallow	2,434	2.43
Google Image Bot Disallow	2,150	2.15
Sogou Bot Disallow	2,145	2.14
Google AdsBot Mobile Disallow	2,117	2.12
Diffbot Disallow	1,706	1.71
Exalead Bot Disallow	1,645	1.64
Google News Bot Disallow	1,476	1.48
Google AdsBot Mobile Apps Disallow	1,353	1.35
Soso Bot Disallow	1,333	1.33
DuckDuckGo Bot Disallow	1,203	1.2
360 Bot Disallow	1,066	1.07
YOU Disallow	1,057	1.06
Google Video Bot Disallow	1,004	1
Google Favicon Disallow	820	0.82
Google API Bot Disallow	815	0.82
Google DuplexWeb Disallow	775	0.78
BuiltWithBot Disallow	557	0.56
Pipl Disallow	466	0.47
Youdao Bot Disallow	199	0.2
LLMS Text	35	0.03
Google FeedFetcher Disallow	32	0.03
Google Read Aloud Disallow	17	0.02
Google Web Light Disallow	8	0.01
Google BIG Bench Disallow	3	0