Your robots.txt file decides which crawlers can visit your site. Since AI tools arrived, many sites have added blocks without understanding the difference between crawlers used for search and those used for training. Get it wrong and you can vanish from AI answers without meaning to.
The examples below are illustrative only. Check each company’s current documentation, because crawler names and rules change.
1. Search crawlers vs training crawlers
OpenAI documents separate crawlers: OAI-SearchBot finds pages to show and link to in ChatGPT search, while GPTBot collects content that may be used to train models. Google uses Googlebot for Search, including AI Overviews, and a separate token, Google-Extended, to control whether content is used for Gemini models.
2. A sensible default for most businesses
If you want customers to find you through AI search, allow the search crawlers. You can still block training crawlers if you prefer.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Blocking Google-Extended does not remove you from Google Search or AI Overviews. Blocking Googlebot would.
3. Check you haven’t blocked yourself
Open yoursite.com/robots.txt and look for broad rules such as User-agent: * with Disallow: /, or security plugins and CDNs blocking bots. Also check page-level noindex and nosnippet tags on pages you want cited. My technical SEO audit guide covers these checks.
4. Other AI crawlers you may see
OpenAI and Google are not the only companies crawling the web. You may see Anthropic’s ClaudeBot, PerplexityBot, Applebot and Bingbot in your server logs. Bing matters more than many people realise, because several AI assistants draw on Bing’s index; my Bing Webmaster Tools guide shows how to check your site there.
Each company publishes its own crawler names and rules. Before adding a block, read their current documentation and decide whether that crawler feeds a search product you want to appear in, or only collects training data.
5. How CDNs and security plugins get in the way
Many sites block AI crawlers without anyone deciding to. Some CDNs and hosting firewalls now include one-click “block AI bots” settings, and some WordPress security plugins block unfamiliar user agents by default. These blocks happen before robots.txt is even read, so your file can say “allow” while the crawler is still turned away.
Check your CDN dashboard, firewall rules and plugin settings, and look at server logs or a log analysis tool to confirm the crawlers you want are actually getting 200 responses.
6. A simple review process
- List the AI products you want to appear in, such as ChatGPT search, Google AI Overviews and Bing Copilot.
- Match each one to the crawler it relies on.
- Check robots.txt, CDN settings and page-level tags for each crawler.
- Decide separately whether you are comfortable with training crawlers.
- Record the decision and review it every few months, because names and rules change.
Example: Lomond Bike Repairs, a fictional Glasgow workshop, discovers its hosting firewall blocks every AI crawler. It allows OAI-SearchBot and Bingbot, keeps GPTBot blocked, and checks its logs a week later to confirm the change worked.
7. What robots.txt can’t do
robots.txt only controls crawling. It does not remove pages that are already indexed, and it does not stop other sites quoting or linking to you. If a page must stay out of Google, use a noindex tag and let Google crawl the page so it can see that tag; blocking the page in robots.txt at the same time would hide the instruction.
It also cannot control what AI tools say about you from other sources, such as reviews, news articles and directory listings. Keeping those accurate matters as much as your own file. Finally, remember that robots.txt is public: anyone can read it, so never use it to hide private areas of your site.
Limitations and next step
robots.txt is a request, not a lock; well-behaved crawlers follow it. For the wider strategy, read how to appear in ChatGPT and AI Overviews.
Sources and review notes
- Google Search Central: AI features and your website — eligibility and controls.
- OpenAI: overview of OpenAI crawlers — OAI-SearchBot and GPTBot.
