Robots.txt Usage Distribution in Pakistan

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Ahrefs Bot Disallow	12,125	29.47
Google AdsBot Disallow	11,640	28.29
Googlebot Disallow	2,516	6.12
Robots Disallow	2,375	5.77
Baidu Bot Disallow	1,835	4.46
Some Robots Disallow	1,380	3.35
Yandex Bot Disallow	912	2.22
Microsoft BingBot Disallow	898	2.18
Yahoo Slurp Disallow	840	2.04
Alexa Bot Disallow	760	31.64
Sogou Bot Disallow	713	1.73
Soso Bot Disallow	702	1.71
Google AdsBot Mobile Disallow	622	1.51
Majestic Bot Disallow	499	1.21
Semrush Bot Disallow	472	1.15
Common Crawl Bot Disallow	286	11.91
Google Image Bot Disallow	275	0.67
GPTBot Disallow	255	10.62
Google Media Partners Disallow	178 ^*	0.43
Google Bard Disallow	177	7.37
AppleBot Disallow	176	0.43
Anthropic Claude Bot Disallow	157	6.54
Facebook Bot Disallow	155	6.45
Google AdsBot Mobile Apps Disallow	146	0.35
Apple AI Bot Disallow	140	5.83
Cohere AI Disallow	139	5.79
Exalead Bot Disallow	138	0.34
Perplexity	134	5.58
ByteDance Bytespider Disallow	112	4.66
Microsoft MSNBot Disallow	85	0.21
360 Bot Disallow	72	0.18
BuiltWithBot Disallow	70	0.17
Google News Bot Disallow	27	0.07
Youdao Bot Disallow	26	0.06
Google Video Bot Disallow	25	0.06
Diffbot Disallow	22	0.92
Webz Omgili Disallow	16	0.67
DuckDuckGo Bot Disallow	11	0.03
Google Favicon Disallow	7	0.02
Google API Bot Disallow	7	0.02
Google DuplexWeb Disallow	6	0.01
Pipl Disallow	4	0.01
YOU Disallow	3	0.12
Google FeedFetcher Disallow	1	0