Published 13 November 2026 | By Noleen Thompson, Managing Director and Founder, LadyBugz Marketing
Blocking AI from your website can mean keeping your content out of AI training, or keeping your company out of the answers buyers read. Here’s how to tell them apart, and who should decide.
Key Insights
- Training and answers are separate settings: AI companies run some bots (automated programs that read web pages) to collect training content and others to fetch pages for answers. OpenAI and Anthropic document each kind separately.
- A shared training opt-out is on its way: on 30 October 2026, Google, Microsoft, OpenAI and Anthropic backed one “no AI training” signal in robots.txt, the public file that tells bots which pages they may visit. Anthropic has started recognising it, Microsoft starts on 1 February 2027 and Google plans early 2027.
- Leaving Google’s AI answers takes one setting: since 31 August 2026, any website can leave AI Overviews (the AI summaries above Google results) and AI Mode (Google’s chat-style search) through Google Search Console, Google’s tool for site owners. Google says this doesn’t affect search ranking, but the site gets no traffic from those features.
- Security settings can block search too: since 15 September 2026, the “Block” option for AI training at Cloudflare, a security service many websites sit behind, also shuts out the search crawlers of Google, Microsoft Bing and Apple.
- No special AI file needed: Google says you don’t need “new machine readable files, AI text files, or markup” to appear in its AI features.
Editorial note: correct as at 13 November 2026. The shared training opt-out is only starting to roll out, and we’ll update this page as that changes.
Most B2B (business-to-business) companies shouldn’t block AI from their website outright. Saying no to AI training is a fair business choice, but blocking the bots that fetch pages for AI answers takes your company out of the shortlists buyers now get from ChatGPT, Google and others. Treat them as two decisions, and make each one on purpose.
If you asked me whether to block AI, my first question back would be: block it from doing what? Legal and IT teams have good reason to be wary of AI companies collecting web content. But “block AI” is often one tick box in a security tool, and it can switch off far more than anyone meant to. That’s why our own audit reports training bots and answer bots separately, because the right setting for one is often wrong for the other. As we said in our look at B2B marketing in 2027, AI tools increasingly draw up the first shortlist.
What does “blocking AI” actually block?
“Blocking AI” can block two different kinds of bot visit. Some bots collect pages to train AI models. Others read pages for AI search, so a tool like ChatGPT can name your company in an answer and credit your page. Training shapes what a model knows. Answers decide whether your company is named when a buyer asks.
Most of this is set in robots.txt, which tells crawlers (the bots that read websites) which pages they may visit. OpenAI’s crawler documentation spells out the split. A site can allow OAI-SearchBot, which surfaces websites in ChatGPT search, while blocking GPTBot to signal that its content shouldn’t be used to train OpenAI’s models. Anthropic, the maker of Claude, makes the same split, with ClaudeBot for training and separate bots for search and user questions.
Google works differently. Googlebot crawls for Google Search, and the same crawl feeds AI Overviews and AI Mode. Training for Google’s Gemini models is controlled by Google-Extended, a name you use in robots.txt rather than a separate bot, which Google says doesn’t affect Search ranking.
One catch: visits a person triggers by asking a question don’t always follow robots.txt. OpenAI says that for these, “robots.txt rules may not apply”. Robots.txt tells well-behaved bots what you’d like. It can’t stop the rest.
For a board, I’d put it like this. “Don’t train on us” is one decision. “Don’t mention us” is another, and very few B2B companies want the second. Our article on AI visibility explains why.
A shared “no AI training” signal is on its way
Website owners will soon be able to say “no AI training” to several AI companies with one shared line in robots.txt. On 30 October 2026, Google announced plans to support one training opt-out based on work at the IETF, the Internet Engineering Task Force, which sets many internet standards. Microsoft, OpenAI and Anthropic committed to it the same day.
Instead of a separate line for each company’s bot, the new approach adds one shared line that participating companies agree to read. The proposal, written by an Anthropic engineer, is still a draft, and the start dates differ:
Microsoft: from 1 February 2027, according to the Bing blog.
Google: early 2027, says its crawler documentation.
Anthropic: began recognising the signal on 30 October 2026 and is rolling it out over time, says its help centre.
OpenAI: “in the coming months”, as PPC Land reported from OpenAI’s announcement.
Read the small print. Unless a site names Google-Extended separately, Google plans to treat the shared opt-out as a “no” to Google-Extended. Google-Extended also decides whether Gemini can draw on your pages and cite them in its answers, which Google calls grounding. So at Google, saying no to training could also cost you some visibility in Gemini. Google says sites that allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps”. Microsoft keeps grounding as a separate setting and warns that switching it off means “reduced brand visibility in AI experiences”.
And don’t expect it to reach back in time. Anthropic says blocking its training bot signals that a site’s “future materials” should be excluded, and we haven’t seen any of the others promise to remove content already collected.
What happens if you opt out of Google’s AI answers?
Opting out of Google’s AI answers removes your site from AI Overviews, AI Mode and the AI features in Google Discover (the article feed in the Google app). You then get no traffic or impressions (times your site is shown) from them. Since 31 August 2026, any site owner can do this with the “Search generative AI” setting in Google Search Console.
The setting began as a UK test in June 2026, and Google’s help page says it has since reached all websites worldwide, which includes South African sites. It doesn’t affect AI training.
The cost is still unclear, because few sites have tried it in public. Search consultant Cyrus Shepard opted one site out and shared what he called a “small case study”, reported by Search Engine Roundtable on 30 October 2026. His AI impressions went to zero, and “regular organic clicks ALSO DROPPED”. One site proves nothing, but it’s a reason to measure first. Search Console now reports impressions from AI Overviews and AI Mode, so you can see what you’d give up.
Is your firewall already blocking AI?
Your firewall may already be blocking AI, because security services such as Cloudflare now offer a simple AI block setting, and IT or the web host can switch it on without anyone telling marketing. Google’s guidance on AI features lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” among the basics.
A firewall is a security filter that decides which visitors, people or bots, get through to your website, and a CDN (content delivery network) serves your pages from servers around the world.
On 15 September 2026 Cloudflare split its training control. “Disallow AI Training” asks crawlers not to train on your pages while keeping Google, Microsoft and Apple’s search crawlers working, though Microsoft only reads that request from early 2027. “Block” now stops Googlebot, Bingbot and Applebot completely. Those are the crawlers behind Google, Bing and Apple search. Cloudflare moved existing training blocks to the gentler setting, so the risk is someone picking the wrong option from now on.
What we saw at a client: We saw how hidden these layers can be at a training company we work with. Its robots.txt welcomed every bot everywhere except the admin area. But the security tool at its web host applied its own rules before any page loaded: a cap on how often each kind of bot could visit, and a hidden trap link that turns away whatever follows it. We only found out in late September 2026, when our own automated site tools started getting “rate limit exceeded” messages and then refusals. It took two requests to the host and about two weeks before our tools could work normally again.
The limits weren’t a mistake. They protected a small server from overload. When we checked the site’s server logs (its record of visits) in October, Google’s and Bing’s crawlers were reaching the site normally, and no AI search bot had been turned away. Bing’s crawler had walked into the trap link twice, though, so afterwards we added one line to the robots.txt to steer well-behaved bots away from it. What I took from it is that an open robots.txt only tells you what a site asks for. The security layers marketing may never see decide the rest. It also taught me to ask. Even with a client we have worked with for years, ask them and their web host what sits in front of the website, then check it yourself.
Can you keep your content out of AI training and still be found?
You can mostly keep your content out of AI training and still be found in AI answers, if you treat training and answers as separate switches. Say no to training where you want to, and leave the answer bots, and Google’s own crawler, free to read your site.
That’s the heart of answer engine optimisation (AEO), the work of getting found and quoted by AI tools, and it doesn’t need anything new bolted on. We acted on Google’s guidance ourselves. On 11 November we took llms.txt (a proposed file meant to guide AI tools) off our own AEO service page, and the page now talks about robots.txt access instead.
Who in your company should make this call?
Marketing, IT and legal should decide on blocking AI together, write it down, and review it when the rules change. IT looks after server load and security and legal looks after content rights and personal information, while whether buyers can still find you is marketing’s worry.
Without a clear owner, the decision falls to whoever controls the setting. That’s how a rule meant to stop bots copying content can hide a company from AI search without marketing knowing. A short written decision fixes it: what you allow for training, what you allow for answers, who owns each setting and when you’ll look again. Put early 2027 in the diary.
POPIA, the Protection of Personal Information Act, often comes up here. In September 2026 ITWeb reported Pansy Tlakula, chairperson of the Information Regulator (which enforces POPIA), saying South Africa doesn’t yet have a national AI policy. We haven’t found any Regulator guidance on AI crawlers. Robots.txt is a public request to bots, not a privacy control, so whether personal information belongs on a public page is a question for your legal adviser.
LadyBugz is a marketing agency, not a law firm, and nothing in this article is legal advice.
Frequently asked questions
Does blocking GPTBot remove us from ChatGPT?
Blocking GPTBot doesn’t remove you from ChatGPT search. OpenAI says GPTBot collects content that may be used for training, while OAI-SearchBot finds pages for ChatGPT search, and each setting is independent. Block OAI-SearchBot and OpenAI says your site won’t be shown in ChatGPT search answers, though it can still appear as a plain link to your site.
Does blocking AI hurt our Google ranking?
Google says blocking AI training with Google-Extended, its training control, doesn’t affect your Google ranking, and neither does its Search Console opt-out from AI features. Blocking Googlebot is different, because it stops Google reading your pages for Search and its AI features alike.
Do we need an llms.txt file?
You don’t need an llms.txt file to appear in Google’s AI features. Google’s guidance says no new machine readable files, AI text files or special code on your pages are needed.
Does POPIA require us to block AI crawlers?
As at November 2026, we haven’t found any Information Regulator guidance telling companies to block AI crawlers. POPIA governs how personal information is processed, so if your website carries personal information, ask your own legal adviser how the Act applies. This answer isn’t legal advice.
A word over lunch
My lunch advice would be short. Don’t let a tick box make a board-level decision. Ask what your website and security tools allow today, then decide about training and about answers separately, and put it in writing with a review date in early 2027. For most B2B companies, I’d leave the answer bots alone. An AI tool can’t quote a website it can’t read, and that’s where shortlists start.
Check whether AI search can find you
Our audit reviews your whole B2B marketing strategy across six areas, including whether AI search can find you. A LadyBugz strategist reads every report before it reaches you.
Questions? Contact me at noleen@ladybugz.co.za
Noleen Thompson, Managing Director and Founder, LadyBugz Marketing
Keep reading
