Robots.txt Usage Distribution in Bangladesh

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	1,514	12.08
Robots Disallow	1,377	10.98
Baidu Bot Disallow	1,101	8.78
Some Robots Disallow	825	6.58
Google AdsBot Disallow	757	6.04
Sogou Bot Disallow	753	6.01
Yandex Bot Disallow	632	5.04
Ahrefs Bot Disallow	605	4.83
Alexa Bot Disallow	602	4.8
Microsoft BingBot Disallow	583	4.65
Yahoo Slurp Disallow	561	4.48
Soso Bot Disallow	553	4.41
Google Media Partners Disallow	454 ^*	3.62
Google AdsBot Mobile Disallow	339	2.7
Majestic Bot Disallow	200	1.6
GPTBot Disallow	199	1.59
Semrush Bot Disallow	180	1.44
Common Crawl Bot Disallow	157	1.25
Google Bard Disallow	124	0.99
AppleBot Disallow	111	0.89
Anthropic Claude Bot Disallow	110	0.88
Facebook Bot Disallow	101	0.81
Google AdsBot Mobile Apps Disallow	101	0.81
Apple AI Bot Disallow	95	0.76
Perplexity	93	0.74
Cohere AI Disallow	92	0.73
Google Image Bot Disallow	53	0.42
Microsoft MSNBot Disallow	46	0.37
Exalead Bot Disallow	42	0.34
360 Bot Disallow	28	0.22
ByteDance Bytespider Disallow	27	0.22
Webz Omgili Disallow	15	0.12
Google News Bot Disallow	14	0.11
BuiltWithBot Disallow	13	0.1
Google Video Bot Disallow	13	0.1
DuckDuckGo Bot Disallow	13	0.1
Google Favicon Disallow	12	0.1
Google API Bot Disallow	12	0.1
Google DuplexWeb Disallow	12	0.1
Youdao Bot Disallow	7	0.06
YOU Disallow	5	0.04
Diffbot Disallow	4	0.03
Pipl Disallow	1	0.01