Robots.txt — how to set up the right file for Google and AI bots
Published 2026-09-30 · Updated 2026-09-30
Robots.txt is a text file at the root of your site (yoursite.com/robots.txt) that tells search engines and AI bots which pages they may read. One wrong line can hide you from both Google and ChatGPT — the right file lets you control exactly which AI services use your content.
Free toolCreate robots.txt for free →What is robots.txt?
Robots.txt follows the Robots Exclusion Protocol, which crawlers read before fetching pages. The file is made of groups: a User-agent line naming the bot, followed by Allow and Disallow lines with paths.
Important: robots.txt is an instruction, not a lock. Reputable bots such as Googlebot, GPTBot and PerplexityBot respect it, but it does not protect sensitive data. Use authentication for that.
Where do I find robots.txt?
It always lives at /robots.txt directly under the domain, e.g. https://yoursite.com/robots.txt. Type the address in your browser to view it. If there is no file, every bot may read everything.
WordPress generates a virtual robots.txt that you can edit through an SEO plugin or by uploading your own file. On Shopify you edit it via the theme (robots.txt.liquid).
Which AI bots exist and what do they do?
AI companies use separate bots for separate purposes, so you can appear in AI search while declining model training.
- ▸GPTBot — OpenAI's crawler for training data
- ▸OAI-SearchBot — OpenAI's crawler for ChatGPT search
- ▸ChatGPT-User — fetches pages when a user asks ChatGPT to visit a link
- ▸PerplexityBot — indexes pages for Perplexity answers
- ▸Google-Extended — controls use for Gemini training (does not affect regular Google Search)
- ▸ClaudeBot — Anthropic's crawler
- ▸CCBot — Common Crawl, an open dataset many AI models are trained on
Does your business show up in AI answers?
Free AI Visibility Score — 8 questions run against ChatGPT, Perplexity, Gemini and Google AI Overviews.
Examples: common robots.txt rules
Allow everything: User-agent: * followed by an empty Disallow: line. Block everything (block all / disallow all): User-agent: * and Disallow: /. Never use that on a live site — you will disappear from search.
Block a folder: Disallow: /admin/. Block AI training but allow AI search: add a group with User-agent: GPTBot and Disallow: /, while leaving OAI-SearchBot and PerplexityBot allowed.
Also add Sitemap: https://yoursite.com/sitemap.xml so bots find all your pages.
Common mistakes that make you invisible
We see these often when analysing AI visibility:
- ▸Disallow: / left over from staging
- ▸All unknown bots blocked, which also stops AI search crawlers
- ▸CSS and JavaScript folders blocked so pages can't be rendered
- ▸Firewall or CDN blocking AI bots even though robots.txt allows them
- ▸No Sitemap line
How do I test my robots.txt?
Google Search Console shows how Google reads your robots.txt. For AI bots, use our free AI bot access checker: enter your URL and we read the file and show which AI crawlers are allowed or blocked.
Checklist
- 01Open yoursite.com/robots.txt and confirm it exists
- 02Make sure there's no Disallow: / under User-agent: *
- 03Decide which AI bots may read the site (search vs training)
- 04Allow OAI-SearchBot and PerplexityBot to appear in AI search
- 05Add a Sitemap line
- 06Test with the AI bot checker and Google Search Console
Frequently asked questions
What is robots.txt?
A text file at yoursite.com/robots.txt that tells search engines and AI bots which parts of the site they may read.
Does robots.txt affect whether I appear in ChatGPT?
Yes. Blocking OAI-SearchBot stops ChatGPT search from reading your pages. Blocking GPTBot stops training, not necessarily search.
Does Google-Extended remove my site from Google?
No. Google-Extended only controls use for Gemini models. Regular search is controlled by Googlebot.
How do I block all bots?
User-agent: * followed by Disallow: /. It removes the site from search and AI search, so only use it for staging.
Is robots.txt a security measure?
No. The file is public and relies on bots complying voluntarily. Protect sensitive content with authentication.
Does your business show up in AI answers?
Free AI Visibility Score — 8 questions run against ChatGPT, Perplexity, Gemini and Google AI Overviews.