Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution in the Top 1 Million Sites
Technology Websites %
Robots Disallow 163,669 16.37
Ahrefs Bot Disallow 69,476 6.95
Googlebot Disallow 57,398 5.74
Google AdsBot Disallow 48,482 4.85
Yandex Bot Disallow 44,307 4.43
Some Robots Disallow 41,977 4.2
Semrush Bot Disallow 33,897 3.39
Baidu Bot Disallow 33,893 3.39
GPTBot Disallow 33,651 3.37
Majestic Bot Disallow 33,459 3.35
Common Crawl Bot Disallow 25,450 2.54
Microsoft BingBot Disallow 24,860 2.49
Anthropic Claude Bot Disallow 21,387 2.14
AppleBot Disallow 20,992 2.1
Google Media Partners Disallow 20,621 2.06
Sogou Bot Disallow 19,904 1.99
Alexa Bot Disallow 18,877 1.89
Yahoo Slurp Disallow 18,858 1.89
Google Bard Disallow 15,940 1.59
Facebook Bot Disallow 14,223 1.42
Microsoft MSNBot Disallow 13,762 1.38
ByteDance Bytespider Disallow 12,936 1.29
Google AdsBot Mobile Disallow 12,931 1.29
Perplexity 11,571 1.16
Google Image Bot Disallow 11,292 1.13
Soso Bot Disallow 10,895 1.09
Cohere AI Disallow 10,774 1.08
Apple AI Bot Disallow 9,895 0.99
Exalead Bot Disallow 9,759 0.98
Webz Omgili Disallow 9,727 0.97
Google AdsBot Mobile Apps Disallow 8,442 0.84
Google News Bot Disallow 7,193 0.72
DuckDuckGo Bot Disallow 6,461 0.65
360 Bot Disallow 6,297 0.63
Diffbot Disallow 5,995 0.6
Google Video Bot Disallow 5,759 0.58
Google Favicon Disallow 5,115 0.51
Google API Bot Disallow 5,022 0.5
Google DuplexWeb Disallow 4,904 0.49
YOU Disallow 4,300 0.43
BuiltWithBot Disallow 2,838 0.28
Pipl Disallow 2,427 0.24
Youdao Bot Disallow 918 0.09
Google FeedFetcher Disallow 142 0.01
LLMS Text 133 0.01
Google Read Aloud Disallow 58 0.01
Google Web Light Disallow 15 0
Google BIG Bench Disallow 5 0

940,887 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 07th Mar 2025.

Google Media Partners Disallow Google Media Partners Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
940,887
Top 100k
Top 100,000 sites by traffic
175,918
Top 10k
Top 10,000 sites by traffic
26,052
Entire Internet
169,555,244
Canada
1,722,614
United States
40,249,603
Germany
6,601,225
United Kingdom
3,778,148
Russia
3,306,770
Australia
2,462,672
France
1,218,775
Italy
1,163,188
Brazil
1,113,026
Netherlands
883,672
Spain
850,031
.co
848,103
China
810,734
Switzerland
772,056
India
744,051
Japan
685,685
PL Poland
463,106
BE Belgium
389,051
SE Sweden
376,786
NZ New Zealand
355,268
ME .me
353,984
AT Austria
332,352
CC Cocos Islands
308,319
CZ Czech Republic
281,617
NO Norway
281,048
MX Mexico
261,076
DK Denmark
253,470
IO .io
252,815
ZA South Africa
233,913
RO Romania
218,026
KR South Korea
216,664
IE Ireland
214,296
TR Turkey
202,209
ID Indonesia
183,296
FI Finland
163,839
PH Philippines
137,301
PT Portugal
134,297
HU Hungary
129,159
TW Taiwan
112,788
AR Argentina
106,284
SG Singapore
104,395
TV .tv
100,157
AI Anguilla
99,136
CL Chile
94,747
GR Greece
94,488
UA Ukraine
88,076
HK Hong Kong
85,890
VN Vietnam
80,735
TH Thailand
78,079
SK Slovakia
74,343
IL Israel
68,503
AE UAE
67,023
MY Malaysia
58,817
VG British Virgin Islands
55,277
LU Luxembourg
53,143
PK Pakistan
39,371
SA Saudi Arabia
37,560
WS Samoa
37,066
LT Lithuania
36,117
BG Bulgaria
32,476
SI Slovenia
31,540
IR Iran
30,920
EE Estonia
30,657
CM Cameroon
29,864
LV Latvia
29,505
PE Peru
28,769
IS Iceland
28,584
IM Isle of Man
20,716
HR Croatia
19,681
NU Niue
18,774
PW Palau
18,161
RS Serbia
17,635
NG Nigeria
17,156
PR Puerto Rico
16,958
CR Costa Rica
16,874
EC Ecuador
16,486
KE Kenya
16,262
BY Belarus
15,644
RE Reunion
14,673
LI Liechtenstein
14,072
TO Tonga
13,817
UY Uruguay
13,815
KZ Kazakhstan
13,018
BD Bangladesh
12,793
SH Saint Helena
12,600
GG Guernsey
12,405
VC Saint Vincent and the Grenadines
12,197
DO Dominican Republic
11,594
MA Morocco
10,733
MT Malta
10,018
EG Egypt
9,929
GT Guatemala
8,993
PM Saint Pierre and Miquelon
8,239
BZ Belize
7,913
AL Albania
7,487
AM Armenia
7,437
LB Lebanon
7,249
PA Panama
7,206
SO Somalia
7,115
GE Georgia
7,081
CY Cyprus
7,013
VE Venezuela
6,964
LK Sri Lanka
6,957
UG Uganda
6,698
FM Micronesia
6,047
UZ Uzbekistan
5,964
QA Qatar
5,882
GH Ghana
5,835
CX Christmas Island
5,194
TZ Tanzania
5,045
KW Kuwait
4,772
NP Nepal
4,760
MD Moldova
4,334
MK Macedonia
4,309
JO Jordan
4,090
HN Honduras
3,852
ET Ethiopia
3,773
PY Paraguay
3,516
MN Mongolia
3,297
TN Tunisia
3,235
AG Antigua and Barbuda
3,231
MU Mauritius
3,212
GP Guadeloupe
3,159
AZ Azerbaijan
2,963
SV El Salvador
2,902
BH Bahrain
2,859
KH Cambodia
2,843
DZ Algeria
2,838
BO Bolivia
2,780
BS Bahamas
2,777
RW Rwanda
2,743
ST Sao Tome and Principe
2,687
MC Monaco
2,678
OM Oman
2,664
BA Bosnia and Herzegovina
2,584
JE Jersey
2,174
KY Cayman Islands
2,151
NA Namibia
2,056
FO Faroe Islands
2,006
VI United States Virgin Islands
1,981
SC Seychelles
1,971
ZW Zimbabwe
1,971
AF Afghanistan
1,934
TT Trinidad and Tobago
1,924
SX Sint Maarten
1,867
ZM Zambia
1,828
JM Jamaica
1,825
LC Saint Lucia
1,753
BM Bermuda
1,736
CI Cote d'Ivoire
1,687
MQ Martinique
1,642
MS Montserrat
1,572
AD Andorra
1,497
YT Mayotte
1,467
IQ Iraq
1,453
MV Maldives
1,445
NI Nicaragua
1,421
AS American Samoa
1,419
MZ Mozambique
1,390
MM Myanmar
1,383
GD Grenada
1,328
KG Kyrgyzstan
1,298
BB Barbados
1,296
NC New Caledonia
1,265
AO Angola
1,255
SN Senegal
1,226
GI Gibraltar
1,117
GY Guyana
1,098
CD Congo
1,071
GL Greenland
1,054
MG Madagascar
1,018
BW Botswana
1,008
TC Turks and Caicos Islands
986
FJ Fiji
965
GS South Georgia and the South Sandwich Islands
896
HT Haiti
848
AW Aruba
818
PS Palestine
774
WF Wallis and Futuna
696
BF Burkina Faso
689
TG Togo
635
TJ Tajikistan
572
CU Cuba
568
GM Gambia
553
GF French Guiana
544
ML Mali
534
MW Malawi
515
VU Vanuatu
510
BJ Benin
464
NE Niger
403
SB Solomon Islands
395
LR Liberia
394
BN Brunei
374
DJ Djibouti
371
SL Sierra Leone
343
PG Papua New Guinea
326
SR Suriname
325
BI Burundi
309
SY Syria
308
AX Åland Islands
304
SD Sudan
297
GU Guam
293
TL East Timor
284
YE Yemen
277
LA Lao People's Democratic Republic
268
PF French Polynesia
236
LY Libya
219
TM Turkmenistan
219
HM Heard Island and McDonald Islands
208
BT Bhutan
181
SM San Marino
175
LS Lesotho
164
SZ Swaziland
164
KN Saint Kitts and Nevis
148
MR Mauritania
148
CW Curacao
142
CK Cook Islands
138
GA Gabon
133
MO Macao
111
NF Norfolk Island
103
MP Northern Mariana Islands
99
SJ Svalbard and Jan Mayen
94
DM Dominica
90
UM United States Minor Outlying Islands
83
GN Guinea
73
CG Congo
69
SS South Sudan
54
CV Cabo Verde
50
TD Chad
46
FK Falkland Islands
40
PN Pitcairn
40
KI Kiribati
34
GW Guinea-Bissau
28
AQ Antarctica
27
GQ Equatorial Guinea
24
MH Marshall Islands
23
BL Saint Barthélemy
20
BQ Bonaire
20
NR Nauru
19
VA Vatican City
19
Robots.txt Categories