Robots.txt Usage Distribution in Pitcairn

Top In Robots.txt Usage Distribution in Pitcairn

Technology	Websites	%
Robots Disallow	8	19.51
Googlebot Disallow	4	9.76
Yandex Bot Disallow	4	9.76
Alexa Bot Disallow	2	11.11
GPTBot Disallow	2	11.11
Common Crawl Bot Disallow	2	11.11
AppleBot Disallow	2	4.88
Anthropic Claude Bot Disallow	2	11.11
Google Bard Disallow	2	11.11
Facebook Bot Disallow	2	11.11
Cohere AI Disallow	2	11.11
Apple AI Bot Disallow	2	11.11
Perplexity	2	11.11
Google AdsBot Mobile Apps Disallow	2	4.88
Microsoft BingBot Disallow	1	2.44
Yahoo Slurp Disallow	1	2.44
Soso Bot Disallow	1	2.44

41 Detections

of Robots.txt in Pitcairn. Last updated 21st Apr 2025.

Robots Disallow is currently the most popular technology in this category.