Robots.txt Nutzungsverteilung In Israel

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	20.718	24,23
Googlebot Disallow	18.844	22,04
Google AdsBot Mobile Disallow	18.230	21,32
Ahrefs Bot Disallow	4.133	4,83
Robots Disallow	3.217	3,76
Majestic Bot Disallow	2.206	2,58
Semrush Bot Disallow	1.570	1,84
Yandex Bot Disallow	1.287	1,51
Common Crawl Bot Disallow	973	1,14
GPTBot Disallow	966	1,13
AppleBot Disallow	904	1,06
Anthropic Claude Bot Disallow	886	1,04
Google Bard Disallow	869	1,02
Baidu Bot Disallow	866	1,01
Apple AI Bot Disallow	857	1
Facebook Bot Disallow	849	0,99
Perplexity	822	0,96
Cohere AI Disallow	805	0,94
Google AdsBot Mobile Apps Disallow	794	0,93
Microsoft BingBot Disallow	751	0,88
Some Robots Disallow	740	0,87
Alexa Bot Disallow	669	0,78
Soso Bot Disallow	640	0,75
Sogou Bot Disallow	609	0,71
Yahoo Slurp Disallow	606	0,71
Exalead Bot Disallow	368	0,43
Google Media Partners Disallow	321 ^*	0,38
Microsoft MSNBot Disallow	196	0,23
Google Image Bot Disallow	195	0,23
Webz Omgili Disallow	156	0,18
YOU Disallow	104	0,12
360 Bot Disallow	85	0,1
ByteDance Bytespider Disallow	72	0,08
BuiltWithBot Disallow	49	0,06
DuckDuckGo Bot Disallow	26	0,03
Diffbot Disallow	25	0,03
Google News Bot Disallow	24	0,03
Google Video Bot Disallow	21	0,02
Google Favicon Disallow	21	0,02
Google API Bot Disallow	13	0,02
Google DuplexWeb Disallow	13	0,02
Pipl Disallow	7	0,01
Youdao Bot Disallow	6	0,01
Google Read Aloud Disallow	1	0