로봇.txt 사용량 분포 ~에 네덜란드

핀을 꼽는 로봇.txt 사용량 분포 ~에 네덜란드

기술	웹사이트	%
Googlebot Disallow	114,900	11.19
Google AdsBot Disallow	98,612	9.6
Baidu Bot Disallow	71,484	6.96
Ahrefs Bot Disallow	68,098	6.63
Robots Disallow	67,046	6.53
Google AdsBot Mobile Disallow	62,071	6.04
Majestic Bot Disallow	47,122	4.59
Yandex Bot Disallow	43,509	4.24
Some Robots Disallow	39,561	3.85
Semrush Bot Disallow	31,462	3.06
Alexa Bot Disallow	28,396	13.87
Microsoft BingBot Disallow	27,917	2.72
Yahoo Slurp Disallow	27,171	2.65
Sogou Bot Disallow	24,993	2.43
GPTBot Disallow	24,658	12.04
Soso Bot Disallow	23,499	2.29
Common Crawl Bot Disallow	22,791	11.13
AppleBot Disallow	22,427	2.18
Anthropic Claude Bot Disallow	20,506	10.01
Google Bard Disallow	19,117	9.33
Facebook Bot Disallow	18,224	8.9
Google AdsBot Mobile Apps Disallow	17,853	1.74
Cohere AI Disallow	16,673	8.14
Apple AI Bot Disallow	16,671	8.14
Perplexity	16,449	8.03
Exalead Bot Disallow	11,772	1.15
ByteDance Bytespider Disallow	8,214	4.01
Google Image Bot Disallow	7,971	0.78
Microsoft MSNBot Disallow	7,283	0.71
Google Media Partners Disallow	3,005 *	0.29
360 Bot Disallow	2,775	0.27
Webz Omgili Disallow	2,677	1.31
DuckDuckGo Bot Disallow	2,214	0.22
Google News Bot Disallow	1,582	0.15
Google Video Bot Disallow	1,467	0.14
BuiltWithBot Disallow	1,337	0.13
Google Favicon Disallow	1,229	0.12
Google API Bot Disallow	1,114	0.11
Google DuplexWeb Disallow	1,105	0.11
Diffbot Disallow	939	0.46
YOU Disallow	445	0.22
Pipl Disallow	358	0.03
Youdao Bot Disallow	104	0.01
Google FeedFetcher Disallow	21	0
Google Read Aloud Disallow	8	0
Google Web Light Disallow	6	0

1,026,836 탐지

~의 로봇.txt ~에 네덜란드. 마지막 업데이트 01st 11 2024.

Googlebot Disallow 현재 이 카테고리에서 가장 인기 있는 기술입니다.