Robots.txt Usage Distribution in Hungary

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	12,832	12.07
Ahrefs Bot Disallow	10,115	9.52
Googlebot Disallow	9,712	9.14
Robots Disallow	9,592	9.02
Google AdsBot Disallow	9,477	8.92
Google AdsBot Mobile Disallow	6,862	6.46
Semrush Bot Disallow	6,547	6.16
Yandex Bot Disallow	4,538	4.27
Baidu Bot Disallow	4,232	3.98
Alexa Bot Disallow	2,939	2.76
GPTBot Disallow	2,468	2.32
Microsoft BingBot Disallow	2,350	2.21
Common Crawl Bot Disallow	2,286	2.15
Anthropic Claude Bot Disallow	2,042	1.92
Some Robots Disallow	1,897	1.78
AppleBot Disallow	1,665	1.57
Exalead Bot Disallow	1,655	1.56
Sogou Bot Disallow	1,346	1.27
Yahoo Slurp Disallow	1,267	1.19
Google Bard Disallow	1,238	1.16
Soso Bot Disallow	1,181	1.11
Facebook Bot Disallow	1,160	1.09
Google AdsBot Mobile Apps Disallow	1,060	1
Apple AI Bot Disallow	1,053	0.99
Cohere AI Disallow	989	0.93
Perplexity	960	0.9
ByteDance Bytespider Disallow	891	0.84
Google Media Partners Disallow	743 ^*	0.7
Google Image Bot Disallow	701	0.66
Microsoft MSNBot Disallow	577	0.54
360 Bot Disallow	298	0.28
Webz Omgili Disallow	285	0.27
Google News Bot Disallow	256	0.24
BuiltWithBot Disallow	233	0.22
Google Video Bot Disallow	233	0.22
DuckDuckGo Bot Disallow	152	0.14
Google Favicon Disallow	116	0.11
Google API Bot Disallow	113	0.11
Google DuplexWeb Disallow	108	0.1
Diffbot Disallow	71	0.07
YOU Disallow	26	0.02
Pipl Disallow	15	0.01
Google FeedFetcher Disallow	10	0.01
Youdao Bot Disallow	8	0.01
Google Read Aloud Disallow	1	0