Robots.txt Répartition des utilisations en Malaisie

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Google AdsBot Disallow	9 427	16,76
Googlebot Disallow	8 871	15,77
Google AdsBot Mobile Disallow	6 150	10,93
Robots Disallow	4 161	7,4
Ahrefs Bot Disallow	2 933	5,21
Baidu Bot Disallow	2 293	4,08
Sogou Bot Disallow	2 062	3,67
Yandex Bot Disallow	1 641	2,92
Some Robots Disallow	1 629	2,9
Alexa Bot Disallow	1 596	19,12
Microsoft BingBot Disallow	1 568	2,79
Yahoo Slurp Disallow	1 455	2,59
Soso Bot Disallow	1 377	2,45
GPTBot Disallow	921	11,03
Anthropic Claude Bot Disallow	879	10,53
Google Image Bot Disallow	877	1,56
Common Crawl Bot Disallow	807	9,67
AppleBot Disallow	789	1,4
Google Bard Disallow	757	9,07
Facebook Bot Disallow	736	8,82
Apple AI Bot Disallow	712	8,53
Perplexity	709	8,49
Cohere AI Disallow	699	8,37
Google AdsBot Mobile Apps Disallow	670	1,19
Majestic Bot Disallow	665	1,18
Semrush Bot Disallow	568	1,01
Google Media Partners Disallow	491 ^*	0,87
Exalead Bot Disallow	150	0,27
Microsoft MSNBot Disallow	125	0,22
ByteDance Bytespider Disallow	118	1,41
Webz Omgili Disallow	93	1,11
Diffbot Disallow	53	0,63
360 Bot Disallow	34	0,06
YOU Disallow	31	0,37
Google News Bot Disallow	28	0,05
Google Video Bot Disallow	28	0,05
DuckDuckGo Bot Disallow	28	0,05
Google Favicon Disallow	24	0,04
Google API Bot Disallow	22	0,04
Google DuplexWeb Disallow	21	0,04
Youdao Bot Disallow	17	0,03
BuiltWithBot Disallow	16	0,03
Pipl Disallow	10	0,02
LLMS Text	1	0