AI, WordPress ·
Should You Block AI Crawlers on Your Website?
By Manu Mathew, Technical Lead Engineer · Published 26 October 2026 · Last updated 26 October 2026
A client looks through their server logs, or a security plugin report, and sees names they don’t recognise: GPTBot, ClaudeBot, OAI-SearchBot. The question follows quickly. Should we block AI crawlers, or let them in?
There is no single right answer, because these bots don’t all do the same job: some collect pages to train AI models, while others help AI search tools find and quote your site. Blocking the wrong one can make your business harder to find, so it helps to know which is which before you change anything.
Quick answer
Most small business sites should not block AI crawlers wholesale. AI companies use separate bots for model training and for search, and robots.txt can treat each one differently, so if you want to keep content out of training, block the training bots only. Keep search bots allowed so AI answer tools can still find and cite you.
Which AI crawlers visit your website, and what do they do?
The main AI companies each run several bots, and they fall into three groups: training, search, and fetching a page because a user asked.
The bot names below come from each company’s own documentation, checked in October 2026. Names change, so check the current lists before you rely on them.
| Purpose | Examples | What blocking does |
|---|---|---|
| Model training | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) | Asks that your content is not used to train future models |
| AI search | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) | Can remove your site from that tool’s search answers |
| User-requested visits | ChatGPT-User (OpenAI), Claude-User (Anthropic) | Stops the tool fetching your page when someone asks about it |
OpenAI’s crawler documentation says each setting is independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot for training. Anthropic describes the same split for its three bots.
Does blocking AI crawlers affect your Google rankings?
Blocking AI training bots doesn’t affect your normal Google rankings, but blocking Googlebot itself would remove you from Google Search.
Google-Extended is not a separate crawler but a control token. Google’s list of common crawlers says it manages whether content may be used for training future Gemini models and for grounding answers in some Gemini products. It also says Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal.
Be careful here. Google’s AI Overviews are part of Google Search, so they rely on Googlebot. Blocking Googlebot to avoid AI Overviews would also take you out of ordinary search results, which few businesses want.
What do you lose if you block AI crawlers?
If you block AI search bots, AI answer tools may stop mentioning or linking to your business.
More people now ask an AI tool questions such as “who can fix my WordPress site in Kozhikode?” instead of typing a search. Those tools can only recommend pages they are allowed to read. OpenAI, for example, says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers.
Blocking training bots costs less in the short term. Your pages stay out of future training data, and search visibility through the search bots is unaffected. The trade-off is less clear-cut, and reasonable owners decide differently.
When does it make sense to block AI crawlers?
Blocking AI training bots makes sense when your content itself is what you sell, or when you have clear legal or client reasons to limit reuse.
Some examples where blocking training bots is a reasonable choice:
- Paid or original content, such as courses, research or detailed guides that are your product.
- Client or contract terms that limit how material may be reused.
- Heavy bot traffic that is slowing a small server, though a crawl-rate limit or caching may fix that without a full block.
For a typical business site, the aim is usually to be found, so allowing search bots is the sensible default.
How do you block AI crawlers with robots.txt?
You add rules to the robots.txt file at the root of your site, naming each bot you want to block.
Here is a setup that blocks the training bots but keeps search and user-requested visits allowed:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpA few things to know:
- WordPress creates a virtual robots.txt if no file exists. Most SEO plugins let you edit it from the dashboard, or you can upload a real file.
- robots.txt is a request, not a lock. The standard, RFC 9309, says these rules are not a form of access authorisation. The bots listed above say they respect it; bad actors may not.
- Changes take time. OpenAI says its search systems can take about a day to pick up a robots.txt change.
- User-requested fetches are a grey area. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, because a person started the request.
If you also want AI tools to understand your site better, my post on what llms.txt is and whether you need one covers that separate file.
How do I help a client decide on AI crawlers?
I start by looking at what the site is for, then write down the options and the trade-offs before changing anything.
- Check the logs. I look at which bots actually visit and how often, so the decision is based on real traffic.
- Agree the goal. Being found, protecting content, or reducing server load each lead to different rules.
- Write the rules down. A short note in plain language records what is blocked and why, so the next person understands it.
- Test and review. I confirm the live robots.txt is valid, then revisit it every few months, because bot names and policies change.
This is the same habit I use at work: make a small change, record the reason, and check the result.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT?
Not on its own. GPTBot is OpenAI’s training crawler, while ChatGPT search uses a separate bot, OAI-SearchBot. If you block GPTBot but allow OAI-SearchBot, OpenAI says your site can still appear in ChatGPT search answers while your content is marked as not for training.
Does Google-Extended stop AI Overviews using my site?
No. Google says Google-Extended controls use of content for training Gemini models and for grounding in some Gemini products, not Google Search. AI Overviews are part of Google Search, so they follow your Googlebot settings, not Google-Extended.
Is robots.txt enough to stop AI bots?
It is enough for bots that follow the standard, and the major AI companies say theirs do. It is not a security control, so a bot that ignores it can still read public pages. For stronger limits you need server or firewall rules, which take more care to set up.
Do AI crawlers slow down a small WordPress site?
They can add load, especially on a small server without page caching. Good caching usually handles it. If one bot is visiting too often, a crawl-rate limit or a targeted block is better than blocking every AI bot at once.
How often should I review my robots.txt?
Every few months is a sensible rhythm, and whenever you hear of a new AI tool your customers use. AI companies add and rename bots, so a list written a year ago may miss some or name bots that no longer exist.
Need help with your site and AI search?
If you’d like a second opinion on your robots.txt, or on how your site shows up in search and AI tools, I’m happy to take a look. The WordPress websites page covers the kinds of work I take on.
You can reach me through the contact section or email me at manu@manu.co.in. Tell me a little about your business and the site, and we’ll go from there.