Robots.txt Nutzungsverteilung In Tunesien

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Robots Disallow	481	13,19
Googlebot Disallow	339	9,3
Google AdsBot Disallow	260	7,13
Ahrefs Bot Disallow	243	6,66
Baidu Bot Disallow	183	5,02
Yandex Bot Disallow	181	4,96
Microsoft BingBot Disallow	156	4,28
Sogou Bot Disallow	151	4,14
Soso Bot Disallow	150	4,11
Alexa Bot Disallow	135	3,7
Yahoo Slurp Disallow	126	3,45
Google AdsBot Mobile Disallow	122	3,35
Majestic Bot Disallow	102	2,8
Some Robots Disallow	92	2,52
Semrush Bot Disallow	82	2,25
GPTBot Disallow	79	2,17
Common Crawl Bot Disallow	77	2,11
Anthropic Claude Bot Disallow	67	1,84
Facebook Bot Disallow	63	1,73
Cohere AI Disallow	62	1,7
Exalead Bot Disallow	54	1,48
Microsoft MSNBot Disallow	52	1,43
ByteDance Bytespider Disallow	42	1,15
360 Bot Disallow	40	1,1
Webz Omgili Disallow	40	1,1
BuiltWithBot Disallow	36	0,99
Google Media Partners Disallow	32 ^*	0,88
Google AdsBot Mobile Apps Disallow	31	0,85
Google Bard Disallow	30	0,82
Perplexity	26	0,71
AppleBot Disallow	25	0,69
Apple AI Bot Disallow	24	0,66
Google Image Bot Disallow	19	0,52
Google News Bot Disallow	10	0,27
Google Video Bot Disallow	10	0,27
Google Favicon Disallow	6	0,16
DuckDuckGo Bot Disallow	5	0,14
Google API Bot Disallow	5	0,14
Google DuplexWeb Disallow	5	0,14
Youdao Bot Disallow	2	0,05
Diffbot Disallow	1	0,03
YOU Disallow	1	0,03