Free tool

robots.txt generator for AI search

Generate a robots.txt for free: choose which AI bots (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) may crawl your site. Copy or download instantly.

GPTBot OpenAI · trainingOAI-SearchBot OpenAI (ChatGPT search) · searchChatGPT-User OpenAI (ChatGPT) · user fetchPerplexityBot Perplexity · searchPerplexity-User Perplexity · user fetchClaudeBot Anthropic · trainingClaude-SearchBot Anthropic · searchClaude-User Anthropic · user fetchGoogle-Extended Google (Gemini) · trainingGooglebot Google (AI Overviews) · searchBingbot Microsoft (Copilot) · searchApplebot-Extended Apple Intelligence · trainingMeta-ExternalAgent Meta AI · trainingCCBot Common Crawl · trainingBytespider ByteDance · training

Paths to block (one per line)

Sitemap URL

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Claude-User
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Applebot-Extended
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: Bytespider
Allow: /
Disallow: /admin/
Disallow: /cart/

Frequently asked questions

What is robots.txt?

A text file at the root of your site (yourdomain.com/robots.txt) that tells search engines and AI bots which parts of the site they may crawl.

Should I block GPTBot?

GPTBot collects data to train models. Blocking it does not affect ChatGPT search, which uses OAI-SearchBot. If you want to appear in AI answers, keep the search bots allowed.

Training bots vs search bots?

Training bots gather text for future models. Search bots fetch pages in real time to cite them in answers. Search bots are the ones that drive visibility and traffic.

Where do I put the file?

Upload robots.txt to your website root so it's reachable at https://yourdomain.com/robots.txt.

Are you actually mentioned in AI answers?

Access is step one. Get your free AI Visibility Score across ChatGPT, Perplexity, Gemini and Google.

Run free analysis