Robot.txt Distribuzione dell'utilizzo su tutta Internet

Distribuzione per siti web che utilizzano Robot.txt tecnologie

In cima Robot.txt Distribuzione dell'utilizzo su tutta Internet

Tecnologia	Siti web	%
Some Robots Disallow	39.138.226	23,93
Googlebot Disallow	13.834.398	8,46
Baidu Bot Disallow	10.188.318	6,23
Sogou Bot Disallow	9.557.148	5,84
Google Media Partners Disallow	9.287.522	5,68
Google AdsBot Disallow	8.717.405	5,33
Google AdsBot Mobile Disallow	7.269.025	4,45
Robots Disallow	5.581.000	3,41
Yandex Bot Disallow	5.356.402	3,28
Microsoft BingBot Disallow	5.195.571	3,18
Alexa Bot Disallow	4.883.371	18,48
Yahoo Slurp Disallow	4.749.920	2,9
Soso Bot Disallow	4.645.220	2,84
Ahrefs Bot Disallow	3.935.542	2,41
GPTBot Disallow	2.727.265	10,32
Common Crawl Bot Disallow	2.682.292	10,15
AppleBot Disallow	2.638.484	1,61
Anthropic Claude Bot Disallow	2.548.782	9,64
Google Bard Disallow	2.421.613	9,16
Facebook Bot Disallow	2.391.535	9,05
Cohere AI Disallow	2.305.690	8,72
Apple AI Bot Disallow	2.273.964	8,6
Perplexity	2.230.267	8,44
Google AdsBot Mobile Apps Disallow	2.114.262	1,29
Majestic Bot Disallow	1.938.772	1,19
Semrush Bot Disallow	1.711.598	1,05
ByteDance Bytespider Disallow	641.870	2,43
Exalead Bot Disallow	395.923	0,24
Microsoft MSNBot Disallow	351.362	0,21
Webz Omgili Disallow	347.921	1,32
360 Bot Disallow	288.753	0,18
Google Image Bot Disallow	264.115	0,16
Diffbot Disallow	179.949	0,68
BuiltWithBot Disallow	156.116	0,1
YOU Disallow	105.454	0,4
Google News Bot Disallow	78.385	0,05
Google Video Bot Disallow	75.967	0,05
DuckDuckGo Bot Disallow	75.846	0,05
Google Favicon Disallow	65.330	0,04
Youdao Bot Disallow	64.292	0,04
Google API Bot Disallow	45.937	0,03
Google DuplexWeb Disallow	43.852	0,03
Pipl Disallow	15.760	0,01
Google FeedFetcher Disallow	3.223	0
Google Read Aloud Disallow	2.071	0
LLMS Text	1.288	0
Google Web Light Disallow	509	0
Google BIG Bench Disallow	24	0