Robots.txt Distribuzione dell'utilizzo In Regno Unito

Distribuzione per i siti Web che utilizzano Robots.txt tecnologie

Tecnologia	Siti web	%
Google AdsBot Disallow	620.380	13,07
Googlebot Disallow	567.242	11,95
Google AdsBot Mobile Disallow	485.522	10,23
Baidu Bot Disallow	245.035	5,16
GPTBot Disallow	187.326	12,84
Common Crawl Bot Disallow	181.947	12,47
Some Robots Disallow	170.310	3,59
Anthropic Claude Bot Disallow	166.307	11,4
Google Bard Disallow	164.754	11,29
Google AdsBot Mobile Apps Disallow	162.899	3,43
Ahrefs Bot Disallow	162.503	3,42
Facebook Bot Disallow	158.006	10,83
AppleBot Disallow	156.609	3,3
Apple AI Bot Disallow	153.848	10,55
Cohere AI Disallow	151.371	10,38
Perplexity	151.009	10,35
Robots Disallow	108.564	2,29
Yandex Bot Disallow	104.244	2,2
Microsoft BingBot Disallow	101.705	2,14
Alexa Bot Disallow	101.682	6,97
Yahoo Slurp Disallow	94.983	2
Sogou Bot Disallow	87.924	1,85
Soso Bot Disallow	86.619	1,83
Majestic Bot Disallow	51.460	1,08
Semrush Bot Disallow	45.819	0,97
ByteDance Bytespider Disallow	11.737	0,8
Microsoft MSNBot Disallow	10.362	0,22
Google Image Bot Disallow	8.968	0,19
Exalead Bot Disallow	8.913	0,19
Webz Omgili Disallow	8.835	0,61
Google Media Partners Disallow	8.626 ^*	0,18
360 Bot Disallow	3.543	0,07
Google News Bot Disallow	2.761	0,06
Google Video Bot Disallow	2.297	0,05
Youdao Bot Disallow	1.867	0,04
Diffbot Disallow	1.810	0,12
Google Favicon Disallow	1.730	0,04
DuckDuckGo Bot Disallow	1.639	0,03
BuiltWithBot Disallow	1.637	0,03
Google API Bot Disallow	939	0,02
Google DuplexWeb Disallow	847	0,02
YOU Disallow	686	0,05
Pipl Disallow	362	0,01
Google FeedFetcher Disallow	44	0
Google Read Aloud Disallow	3	0
Google Web Light Disallow	1	0