Robots.txt Generator
Generate a robots.txt file that controls which crawlers can access your site — including AI bots like GPTBot, ClaudeBot, and PerplexityBot.
Allow these crawlers
robots.txt
User-Agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-Agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-Agent: ClaudeBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-Agent: PerplexityBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-Agent: Bingbot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-Agent: Google-Extended
Disallow: /
User-Agent: Meta-ExternalAgent
Disallow: /RankFlo generates robots.txt automatically
AI-friendly robots.txt with support for GPTBot, ClaudeBot, PerplexityBot, and more — built in.
Start free