Robots.txt Usage Distribution in Mozambique

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	254	15.51
Google AdsBot Mobile Disallow	233	14.22
Googlebot Disallow	227	13.86
Robots Disallow	176	10.74
Common Crawl Bot Disallow	56	3.42
Ahrefs Bot Disallow	54	3.3
GPTBot Disallow	54	3.3
Google AdsBot Mobile Apps Disallow	50	3.05
Google Media Partners Disallow	48 ^*	2.93
Anthropic Claude Bot Disallow	45	2.75
Google Bard Disallow	44	2.69
Facebook Bot Disallow	44	2.69
Cohere AI Disallow	44	2.69
AppleBot Disallow	42	2.56
Apple AI Bot Disallow	42	2.56
Perplexity	42	2.56
Majestic Bot Disallow	34	2.08
Semrush Bot Disallow	33	2.01
Yandex Bot Disallow	28	1.71
Microsoft BingBot Disallow	14	0.85
Alexa Bot Disallow	13	0.79
Sogou Bot Disallow	11	0.67
Yahoo Slurp Disallow	10	0.61
Soso Bot Disallow	10	0.61
Exalead Bot Disallow	8	0.49
Microsoft MSNBot Disallow	5	0.31
Google Image Bot Disallow	5	0.31
ByteDance Bytespider Disallow	4	0.24
360 Bot Disallow	3	0.18
BuiltWithBot Disallow	2	0.12
Webz Omgili Disallow	1	0.06
Google News Bot Disallow	1	0.06
Google Video Bot Disallow	1	0.06