robots.txt Generator for AI

Generate a robots.txt file optimized for AI agents. Select which bots to allow or block, configure Content-Signal directives, and download the result.

AI Bots

Choose how each AI bot can access your site. Default means no specific rule is generated for that bot.

GPTBot OpenAI — Model training
OAI-SearchBot OpenAI — ChatGPT search results
ChatGPT-User OpenAI — ChatGPT user browsing
OAI-AdsBot OpenAI — ChatGPT ads validation
ClaudeBot Anthropic — Model training
Claude-SearchBot Anthropic — Claude search results
Claude-User Anthropic — Claude user browsing
PerplexityBot Perplexity — Perplexity search index
Perplexity-User Perplexity — Perplexity user browsing
Google-Extended Google — Gemini training opt-out
Google-CloudVertexBot Google — Vertex AI agents
meta-externalagent Meta — Meta AI training
MistralAI-User Mistral — Le Chat user browsing
DuckAssistBot DuckDuckGo — DuckAssist answers
Applebot-Extended Apple — Apple Intelligence training opt-out
Amazonbot Amazon — Alexa / Rufus
Bytespider ByteDance — TikTok / Doubao training
CCBot Common Crawl — Open crawl corpus
Claude-Web Anthropic — Retired — use Claude-User / Claude-SearchBot
FacebookBot Meta — Retired — use meta-externalagent
cohere-ai Cohere — Never vendor-documented
| |

Content-Signal Directives

Control how AI agents may use your content. These directives are added to the robots.txt global section.

Allow search indexing by AI

Allow use as AI input/context

Allow use for AI training

Additional Settings

seconds

Generated robots.txt

allowed, blocked, bytes 


        
        

          
          
        

      

    


    
    

      
What is robots.txt for AI?

      
robots.txt is a standard file that tells web crawlers which pages they can access. With AI agents becoming a major source of traffic, it is essential to configure robots.txt with specific rules for AI bots like GPTBot, ClaudeBot, and PerplexityBot. Content-Signal directives let you specify how AI agents may use your content for search, input, or training.

    


    
    

      ← Back to Tools