Robots.txt Usage Distribution in Sao Tome and Principe

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	276	11.48
Baidu Bot Disallow	250	10.4
Robots Disallow	219	9.11
Yandex Bot Disallow	164	6.82
Alexa Bot Disallow	162	6.74
Microsoft BingBot Disallow	161	6.7
Yahoo Slurp Disallow	158	6.57
Sogou Bot Disallow	152	6.32
Soso Bot Disallow	152	6.32
Some Robots Disallow	113	4.7
GPTBot Disallow	51	2.12
Google Media Partners Disallow	48 ^*	2
Common Crawl Bot Disallow	43	1.79
Google Bard Disallow	41	1.71
Google AdsBot Disallow	40	1.66
Anthropic Claude Bot Disallow	39	1.62
Ahrefs Bot Disallow	35	1.46
AppleBot Disallow	35	1.46
Facebook Bot Disallow	35	1.46
Webz Omgili Disallow	31	1.29
Apple AI Bot Disallow	28	1.16
Majestic Bot Disallow	24	1
Semrush Bot Disallow	21	0.87
Cohere AI Disallow	18	0.75
Perplexity	17	0.71
ByteDance Bytespider Disallow	14	0.58
Google AdsBot Mobile Apps Disallow	11	0.46
Microsoft MSNBot Disallow	9	0.37
Exalead Bot Disallow	8	0.33
Diffbot Disallow	8	0.33
Google Image Bot Disallow	6	0.25
YOU Disallow	6	0.25
Google News Bot Disallow	4	0.17
Google Video Bot Disallow	4	0.17
DuckDuckGo Bot Disallow	4	0.17
360 Bot Disallow	3	0.12
Google Favicon Disallow	3	0.12
Google API Bot Disallow	3	0.12
Google DuplexWeb Disallow	3	0.12
BuiltWithBot Disallow	2	0.08
Pipl Disallow	2	0.08
Youdao Bot Disallow	1	0.04