Robots.txt Répartition de l'utilisation dans Singapour

Distribution pour les sites Web utilisant Robots.txt les technologies

Technologie	Sites Internet	%
Google AdsBot Disallow	24 730	18,44
Google AdsBot Mobile Disallow	17 935	13,37
Googlebot Disallow	15 675	11,69
Ahrefs Bot Disallow	7 665	5,71
GPTBot Disallow	4 948	3,69
Common Crawl Bot Disallow	4 882	3,64
Anthropic Claude Bot Disallow	4 868	3,63
AppleBot Disallow	4 840	3,61
Google Bard Disallow	4 808	3,58
Facebook Bot Disallow	4 765	3,55
Apple AI Bot Disallow	4 741	3,53
Perplexity	4 679	3,49
Cohere AI Disallow	4 639	3,46
Google AdsBot Mobile Apps Disallow	4 629	3,45
Robots Disallow	3 754	2,8
Baidu Bot Disallow	2 041	1,52
Some Robots Disallow	1 654	1,23
Majestic Bot Disallow	1 545	1,15
Microsoft BingBot Disallow	1 537	1,15
Yandex Bot Disallow	1 508	1,12
Alexa Bot Disallow	1 504	1,12
Sogou Bot Disallow	1 258	0,94
Yahoo Slurp Disallow	1 211	0,9
Soso Bot Disallow	1 028	0,77
Semrush Bot Disallow	912	0,68
Google Media Partners Disallow	551 ^*	0,41
Exalead Bot Disallow	413	0,31
Google Image Bot Disallow	305	0,23
Microsoft MSNBot Disallow	254	0,19
Webz Omgili Disallow	189	0,14
ByteDance Bytespider Disallow	170	0,13
Diffbot Disallow	75	0,06
DuckDuckGo Bot Disallow	59	0,04
Google News Bot Disallow	54	0,04
Google Video Bot Disallow	51	0,04
Google Favicon Disallow	47	0,04
360 Bot Disallow	40	0,03
YOU Disallow	35	0,03
Youdao Bot Disallow	33	0,02
Google API Bot Disallow	25	0,02
Google DuplexWeb Disallow	24	0,02
Pipl Disallow	23	0,02
BuiltWithBot Disallow	18	0,01