Robots.txt Distribuzione dell'utilizzo In .me

Distribuzione per i siti Web che utilizzano Robots.txt tecnologie

Tecnologia	Siti web	%
Googlebot Disallow	42.825	11,42
Baidu Bot Disallow	37.495	10
Some Robots Disallow	24.447	6,52
Yahoo Slurp Disallow	16.996	4,53
Yandex Bot Disallow	16.647	4,44
Microsoft BingBot Disallow	16.646	4,44
Alexa Bot Disallow	16.356	15,86
Google AdsBot Disallow	16.158	4,31
Sogou Bot Disallow	15.925	4,25
Soso Bot Disallow	15.747	4,2
Robots Disallow	15.636	4,17
Google AdsBot Mobile Disallow	14.178	3,78
GPTBot Disallow	11.815	11,46
Common Crawl Bot Disallow	11.296	10,95
AppleBot Disallow	11.158	2,97
Anthropic Claude Bot Disallow	10.482	10,16
Google Bard Disallow	10.478	10,16
Facebook Bot Disallow	10.315	10
Apple AI Bot Disallow	9.857	9,56
Google AdsBot Mobile Apps Disallow	9.617	2,56
Perplexity	8.970	8,7
Cohere AI Disallow	8.867	8,6
Ahrefs Bot Disallow	4.804	1,28
Semrush Bot Disallow	4.481	1,19
Majestic Bot Disallow	3.889	1,04
Webz Omgili Disallow	1.677	1,63
Exalead Bot Disallow	1.574	0,42
360 Bot Disallow	1.344	0,36
ByteDance Bytespider Disallow	978	0,95
Google Media Partners Disallow	824 ^*	0,22
Microsoft MSNBot Disallow	783	0,21
Google Image Bot Disallow	586	0,16
Google News Bot Disallow	318	0,08
Google Video Bot Disallow	304	0,08
Google Favicon Disallow	277	0,07
BuiltWithBot Disallow	249	0,07
DuckDuckGo Bot Disallow	230	0,06
Diffbot Disallow	190	0,18
Google API Bot Disallow	186	0,05
Google DuplexWeb Disallow	182	0,05
YOU Disallow	138	0,13
Pipl Disallow	83	0,02
Youdao Bot Disallow	70	0,02
Google FeedFetcher Disallow	8	0
Google Read Aloud Disallow	1	0
Google Web Light Disallow	1	0
Google BIG Bench Disallow	1	0