机器人.txt 使用分布 在 德国

置顶 机器人.txt 使用分布 在 德国

技术	网站	%
Googlebot Disallow	993,636	12.79
Baidu Bot Disallow	719,764	9.27
Sogou Bot Disallow	589,407	7.59
Yandex Bot Disallow	495,371	6.38
Alexa Bot Disallow	488,526	6.29
Microsoft BingBot Disallow	485,954	6.26
Yahoo Slurp Disallow	478,428	6.16
Soso Bot Disallow	466,757	6.01
Ahrefs Bot Disallow	383,874	4.94
Robots Disallow	313,247	4.03
Some Robots Disallow	312,290	4.02
Majestic Bot Disallow	296,617	3.82
Semrush Bot Disallow	289,868	3.73
Google AdsBot Disallow	276,877	3.56
ByteDance Bytespider Disallow	246,633	3.18
Google AdsBot Mobile Disallow	198,248	2.55
GPTBot Disallow	58,176	0.75
Anthropic Claude Bot Disallow	52,650	0.68
Common Crawl Bot Disallow	51,845	0.67
Google AdsBot Mobile Apps Disallow	51,737	0.67
Google Bard Disallow	44,150	0.57
Facebook Bot Disallow	42,548	0.55
AppleBot Disallow	41,725	0.54
Google Media Partners Disallow	38,857 *	0.5
Cohere AI Disallow	38,644	0.5
Apple AI Bot Disallow	37,864	0.49
Perplexity	36,587	0.47
Google Image Bot Disallow	32,773	0.42
Microsoft MSNBot Disallow	27,217	0.35
DuckDuckGo Bot Disallow	24,351	0.31
Exalead Bot Disallow	23,548	0.3
Google News Bot Disallow	19,807	0.26
Google Video Bot Disallow	19,791	0.25
Google Favicon Disallow	19,505	0.25
Google API Bot Disallow	19,447	0.25
Google DuplexWeb Disallow	19,300	0.25
Webz Omgili Disallow	10,963	0.14
360 Bot Disallow	7,831	0.1
BuiltWithBot Disallow	4,848	0.06
Diffbot Disallow	3,401	0.04
YOU Disallow	2,368	0.03
Pipl Disallow	867	0.01
Youdao Bot Disallow	274	0
Google FeedFetcher Disallow	139	0
Google Read Aloud Disallow	19	0
Google Web Light Disallow	5	0
Google BIG Bench Disallow	4	0
LLMS Text	1	0

7,766,739 检测结果

的 机器人.txt 在 德国. 最近更新时间 01st 2月 2025.

Googlebot Disallow 是目前该类别中最流行的技术。