Robots.txt Distribuzione dell'utilizzo nei primi 10.000 siti

Distribuzione per i siti Web che utilizzano Robots.txt tecnologie

Tecnologia	Siti web	%
Robots Disallow	3.571	35,71
GPTBot Disallow	1.449	14,49
Some Robots Disallow	1.417	14,17
Googlebot Disallow	1.323	13,23
Common Crawl Bot Disallow	1.114	11,14
Ahrefs Bot Disallow	988	9,88
AppleBot Disallow	986	9,86
Anthropic Claude Bot Disallow	971	9,71
Google AdsBot Disallow	851	8,51
Yandex Bot Disallow	763	7,63
ByteDance Bytespider Disallow	733	7,33
Semrush Bot Disallow	713	7,13
Facebook Bot Disallow	705	7,05
Google Bard Disallow	699	6,99
Perplexity	687	6,87
Webz Omgili Disallow	669	6,69
Cohere AI Disallow	630	6,3
Apple AI Bot Disallow	618	6,18
Majestic Bot Disallow	574	5,74
Baidu Bot Disallow	573	5,73
Microsoft BingBot Disallow	571	5,71
Diffbot Disallow	451	4,51
Alexa Bot Disallow	421	4,21
Yahoo Slurp Disallow	325	3,25
Google Media Partners Disallow	307	3,07
Microsoft MSNBot Disallow	302	3,02
Google AdsBot Mobile Disallow	285	2,85
Sogou Bot Disallow	264	2,64
Google Image Bot Disallow	257	2,57
Exalead Bot Disallow	197	1,97
YOU Disallow	188	1,88
Google News Bot Disallow	185	1,85
Google AdsBot Mobile Apps Disallow	171	1,71
Soso Bot Disallow	161	1,61
DuckDuckGo Bot Disallow	142	1,42
360 Bot Disallow	133	1,33
Google Video Bot Disallow	110	1,1
Google API Bot Disallow	78	0,78
Google Favicon Disallow	74	0,74
Google DuplexWeb Disallow	69	0,69
Pipl Disallow	64	0,64
BuiltWithBot Disallow	57	0,57
Youdao Bot Disallow	24	0,24
Google FeedFetcher Disallow	3	0,03