Robots.txt Nutzungsverteilung In São Tomé und Príncipe

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Googlebot Disallow	276	10,1
Robots Disallow	263	9,63
Baidu Bot Disallow	253	9,26
Sogou Bot Disallow	212	7,76
Yandex Bot Disallow	177	6,48
Microsoft BingBot Disallow	174	6,37
Alexa Bot Disallow	170	6,22
Yahoo Slurp Disallow	170	6,22
Soso Bot Disallow	165	6,04
Google Media Partners Disallow	152 ^*	5,56
Some Robots Disallow	114	4,17
GPTBot Disallow	52	1,9
Common Crawl Bot Disallow	45	1,65
Google Bard Disallow	44	1,61
Anthropic Claude Bot Disallow	43	1,57
Facebook Bot Disallow	40	1,46
AppleBot Disallow	39	1,43
Google AdsBot Disallow	37	1,35
Ahrefs Bot Disallow	35	1,28
Webz Omgili Disallow	35	1,28
Apple AI Bot Disallow	33	1,21
Cohere AI Disallow	21	0,77
Majestic Bot Disallow	21	0,77
Semrush Bot Disallow	21	0,77
Google AdsBot Mobile Disallow	19	0,7
Perplexity	19	0,7
ByteDance Bytespider Disallow	17	0,62
Google AdsBot Mobile Apps Disallow	11	0,4
Diffbot Disallow	11	0,4
Microsoft MSNBot Disallow	9	0,33
YOU Disallow	9	0,33
Exalead Bot Disallow	8	0,29
Google Image Bot Disallow	6	0,22
360 Bot Disallow	4	0,15
Google News Bot Disallow	4	0,15
Google Video Bot Disallow	4	0,15
DuckDuckGo Bot Disallow	4	0,15
BuiltWithBot Disallow	3	0,11
Google Favicon Disallow	3	0,11
Google API Bot Disallow	3	0,11
Google DuplexWeb Disallow	3	0,11
Pipl Disallow	2	0,07
Youdao Bot Disallow	1	0,04