Robots.txt Nutzungsverteilung In Tonga

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	2.017	15,52
Baidu Bot Disallow	1.943	14,95
Some Robots Disallow	1.392	10,71
Microsoft BingBot Disallow	963	7,41
Alexa Bot Disallow	947	7,29
Yandex Bot Disallow	942	7,25
Yahoo Slurp Disallow	922	7,09
Sogou Bot Disallow	914	7,03
Soso Bot Disallow	913	7,02
Google Media Partners Disallow	485 ^*	3,73
Robots Disallow	470	3,62
Ahrefs Bot Disallow	135	1,04
Google AdsBot Disallow	113	0,87
Semrush Bot Disallow	112	0,86
Majestic Bot Disallow	102	0,78
GPTBot Disallow	87	0,67
Common Crawl Bot Disallow	66	0,51
Google Bard Disallow	57	0,44
AppleBot Disallow	48	0,37
Anthropic Claude Bot Disallow	47	0,36
Facebook Bot Disallow	47	0,36
Apple AI Bot Disallow	40	0,31
Google AdsBot Mobile Apps Disallow	35	0,27
Cohere AI Disallow	31	0,24
Perplexity	31	0,24
Webz Omgili Disallow	20	0,15
Exalead Bot Disallow	16	0,12
ByteDance Bytespider Disallow	15	0,12
Google Image Bot Disallow	15	0,12
Microsoft MSNBot Disallow	13	0,1
Google News Bot Disallow	8	0,06
DuckDuckGo Bot Disallow	8	0,06
Diffbot Disallow	7	0,05
360 Bot Disallow	6	0,05
Google API Bot Disallow	6	0,05
Google Video Bot Disallow	5	0,04
Google Favicon Disallow	5	0,04
Google DuplexWeb Disallow	4	0,03
YOU Disallow	4	0,03
Pipl Disallow	4	0,03
BuiltWithBot Disallow	3	0,02
Google FeedFetcher Disallow	1	0,01