Robots.txt Usage Distribution in Philippines

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Some Robots Disallow	80,908	48.18
Googlebot Disallow	10,934	6.51
Baidu Bot Disallow	7,919	4.72
Google AdsBot Disallow	6,914	4.12
Sogou Bot Disallow	6,045	3.6
Yandex Bot Disallow	6,024	3.59
Alexa Bot Disallow	5,864	39.27
Microsoft BingBot Disallow	5,835	3.47
Yahoo Slurp Disallow	5,806	3.46
Soso Bot Disallow	5,717	3.4
Robots Disallow	5,417	3.23
Google AdsBot Mobile Disallow	4,200	2.5
Ahrefs Bot Disallow	3,110	1.85
GPTBot Disallow	1,237	8.28
Common Crawl Bot Disallow	1,211	8.11
Anthropic Claude Bot Disallow	1,096	7.34
Google Bard Disallow	1,090	7.3
Google AdsBot Mobile Apps Disallow	1,067	0.64
Facebook Bot Disallow	1,064	7.13
AppleBot Disallow	1,056	0.63
Apple AI Bot Disallow	1,031	6.9
Perplexity	988	6.62
Cohere AI Disallow	987	6.61
Semrush Bot Disallow	466	0.28
Majestic Bot Disallow	426	0.25
Google Media Partners Disallow	372 ^*	0.22
360 Bot Disallow	368	0.22
Microsoft MSNBot Disallow	147	0.09
Google Image Bot Disallow	119	0.07
Exalead Bot Disallow	118	0.07
Webz Omgili Disallow	100	0.67
ByteDance Bytespider Disallow	87	0.58
Google News Bot Disallow	37	0.02
Google Video Bot Disallow	33	0.02
BuiltWithBot Disallow	26	0.02
DuckDuckGo Bot Disallow	25	0.01
Diffbot Disallow	16	0.11
Google Favicon Disallow	16	0.01
Google API Bot Disallow	16	0.01
Google DuplexWeb Disallow	14	0.01
Youdao Bot Disallow	11	0.01
Pipl Disallow	11	0.01
YOU Disallow	5	0.03
Google FeedFetcher Disallow	3	0