Robots.txt Nutzungsverteilung in Trinidad und Tobago

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	243	11,68
Google AdsBot Mobile Disallow	215	10,33
Googlebot Disallow	150	7,21
Robots Disallow	109	5,24
GPTBot Disallow	103	4,95
AppleBot Disallow	103	4,95
Common Crawl Bot Disallow	102	4,9
Google Bard Disallow	102	4,9
Anthropic Claude Bot Disallow	101	4,85
Facebook Bot Disallow	101	4,85
Apple AI Bot Disallow	101	4,85
Cohere AI Disallow	99	4,76
Perplexity	99	4,76
Google AdsBot Mobile Apps Disallow	99	4,76
Baidu Bot Disallow	41	1,97
Yahoo Slurp Disallow	36	1,73
Yandex Bot Disallow	35	1,68
Microsoft BingBot Disallow	31	1,49
Alexa Bot Disallow	29	1,39
Sogou Bot Disallow	27	1,3
Ahrefs Bot Disallow	27	1,3
Soso Bot Disallow	26	1,25
Some Robots Disallow	25	1,2
Google Media Partners Disallow	15 ^*	0,72
Google Image Bot Disallow	10	0,48
Microsoft MSNBot Disallow	9	0,43
Majestic Bot Disallow	8	0,38
Semrush Bot Disallow	8	0,38
Webz Omgili Disallow	5	0,24
Exalead Bot Disallow	3	0,14
Google News Bot Disallow	3	0,14
Google Video Bot Disallow	3	0,14
DuckDuckGo Bot Disallow	3	0,14
Google Favicon Disallow	3	0,14
ByteDance Bytespider Disallow	2	0,1
360 Bot Disallow	1	0,05
Diffbot Disallow	1	0,05
YOU Disallow	1	0,05
Google API Bot Disallow	1	0,05
Google DuplexWeb Disallow	1	0,05