Robots.txt Distribuzione dell'utilizzo In Maurizio

Distribuzione per i siti Web che utilizzano Robots.txt tecnologie

Tecnologia	Siti web	%
Google AdsBot Disallow	692	18,51
Googlebot Disallow	554	14,82
Google AdsBot Mobile Disallow	513	13,72
Robots Disallow	285	7,62
Ahrefs Bot Disallow	222	5,94
Common Crawl Bot Disallow	91	2,43
AppleBot Disallow	91	2,43
Anthropic Claude Bot Disallow	88	2,35
GPTBot Disallow	86	2,3
Facebook Bot Disallow	85	2,27
Apple AI Bot Disallow	85	2,27
Yandex Bot Disallow	84	2,25
Google Bard Disallow	84	2,25
Alexa Bot Disallow	79	2,11
Perplexity	78	2,09
Microsoft BingBot Disallow	75	2,01
Cohere AI Disallow	74	1,98
Google AdsBot Mobile Apps Disallow	73	1,95
Yahoo Slurp Disallow	72	1,93
Soso Bot Disallow	67	1,79
Some Robots Disallow	50	1,34
Google Media Partners Disallow	49 ^*	1,31
Semrush Bot Disallow	47	1,26
Majestic Bot Disallow	25	0,67
Webz Omgili Disallow	14	0,37
ByteDance Bytespider Disallow	13	0,35
Exalead Bot Disallow	10	0,27
Microsoft MSNBot Disallow	8	0,21
Google Image Bot Disallow	8	0,21
DuckDuckGo Bot Disallow	5	0,13
Diffbot Disallow	4	0,11
Google News Bot Disallow	4	0,11
Google Video Bot Disallow	4	0,11
Google Favicon Disallow	4	0,11
360 Bot Disallow	3	0,08
BuiltWithBot Disallow	3	0,08
YOU Disallow	3	0,08
Google API Bot Disallow	3	0,08
Google DuplexWeb Disallow	3	0,08