robots.txt AI Training Opt Out: One Rule for Google, OpenAI and More

Website owners have wanted a simple robots.txt AI training opt out for years. Until now, you had to block each company’s crawler one by one: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. That started to change on 30 October 2026.

Google, Anthropic, OpenAI and Microsoft each said they will honor one shared robots.txt AI training signal. Add one small group to your file, and every company that supports it should treat your content as off limits for training generative AI models.

Quick answer: Add User-Agent: __usage_ai-training followed by Disallow: / to your robots.txt. It tells supporting AI companies not to train generative models on your site. It does not block search crawlers, so your Google and Bing rankings stay safe. Support starts at different times, so keep your old rules for now.

What changed on 30 October

The new signal comes from an Internet Draft called Machine Readable Training Preference Signal, written by Kevin Kelley of Anthropic. It defines a special name, called a policy token, that you place in robots.txt just like a crawler name.

On the same day, all four companies said they would respect it. As a result, one rule can now speak to several AI labs at once.

One honest caveat: this is an individual draft, not an official IETF standard yet. It feeds into the wider IETF work on AI preferences. Also, robots.txt is a request, not a lock. Companies follow it by choice.

The exact robots.txt AI training rule

To opt your whole site out of AI training, add this group:

User-Agent: __usage_ai-training
Disallow: /

If you want to keep one folder open, add an Allow line:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

You can also protect only one section by naming just that path:

User-Agent: __usage_ai-training
Disallow: /members/

A key detail: a User-Agent: * group does not count. The draft says the wildcard must not match this token, so you have to name it directly. If you never mention the token, you have expressed no preference.

When each company starts honoring it

CompanyStatus on 2 November 2026
AnthropicHas started building support; full coverage over the coming months
MicrosoftTakes effect on 1 February 2027
GooglePlans support in early 2027
OpenAISays it will honor the common signal; no firm date yet

Microsoft confirmed its date on the Bing blog. Google’s crawler documentation says it intends to support the token in early 2027. Reports on OpenAI’s timing differ, so watch its official pages.

Will this hurt my SEO?

No. The token covers training, not crawling. In fact, the draft says it describes how content may be used, not whether a bot may fetch it. Googlebot and Bingbot keep indexing your pages as before.

Google also confirms that Google-Extended, its existing AI control, is not a ranking signal and does not affect inclusion in Search. However, there is one Google detail to know. According to Google’s Search Central post, if you opt out with the shared token and do not name Google-Extended, Google will treat it as if you had blocked Google-Extended too. That can also remove your content from grounding in Gemini apps. If you want to appear in Gemini answers, explicitly allow Google-Extended.

If you work on search visibility, our guide to AI SEO tools shows how to track how AI answers cite your site.

Training versus AI answers

Opting out of training is not the same as hiding from AI answers. Training changes what a model learns. Grounding, on the other hand, means an AI tool fetches your page live to answer a question, often with a link back to you.

Microsoft will also honor a separate page level grounding control for Bing from 1 February 2027. For example, a meta tag can say ai-grounding:disallow. Blocking grounding may cost you visibility and clicks from AI answers, so think before you use it.

Microsoft also made a change for old tags. After 1 March 2027, Bing will no longer read NoArchive or NoCache as AI preferences. If you rely on those, review your setup before then. From 15 December 2026, Bing Webmaster Tools also lets those sites set a temporary grounding preference.

Free: AI Tools Cheat Sheet 2026

24 free tools sorted by task, a simple prompt formula and 10 prompts you can copy. We send the PDF straight to your inbox.

Free. No spam. Unsubscribe in one click.

How to add the rule on WordPress

  1. Open your robots.txt. In Yoast SEO, go to Tools, then File editor. You can also use your host’s file manager.
  2. Add the __usage_ai-training group in its own block. Do not merge it into the User-Agent: * group.
  3. Keep your existing rules for GPTBot, Google-Extended and ClaudeBot until each company’s support is live.
  4. Never block Googlebot, Bingbot, OAI-SearchBot or Claude-SearchBot if you want search traffic.
  5. Save, then visit yoursite.com/robots.txt to confirm the new lines appear.

How it compares with the old crawler rules

The old method listed each company’s crawler by name. That approach still works and stays important for now. For instance, disallowing GPTBot remains OpenAI’s training opt out today, while OAI-SearchBot controls ChatGPT search visibility. Anthropic separates ClaudeBot for training from Claude-User and Claude-SearchBot.

The new token is different in two ways. First, it is shared, so one line covers every supporting company. Second, it is about use, not access. Company specific rules can still take priority over it, so a clear GPTBot rule will keep working as written.

Remember too that the signal works going forward only. It does not remove anything AI companies collected before.

Should you use it?

It depends on your goals. News sites, writers and creators who sell original work have strong reasons to opt out. Meanwhile, businesses that want brand mentions in AI tools may prefer to stay open to training and focus on grounding instead.

Whatever you choose, this is a personal decision about your own content. Users can also control how their own chats and data train AI, which we explain in our guide on how to opt out of AI training.

Also read: Samsung TV Privacy Claims: What a Researcher Says and How to Turn Off Tracking

FAQ

How do I block AI training in robots.txt with one rule?

Add a group with User-Agent: __usage_ai-training and Disallow: /. Support depends on each company’s start date.

Will blocking AI training hurt my Google rankings?

No. The token does not affect Googlebot or Bingbot indexing, and Google-Extended is not a ranking signal.

Does User-agent: * block AI training?

No. The draft says wildcard groups do not match the token. You must name it directly.

Do I still need GPTBot and Google-Extended rules?

Yes, for now. Keep them until each company fully supports the shared token.

Does it remove my content from existing AI models?

No. It only applies to future training. Content collected earlier is not removed.

Leave a Comment

AI Prompts That Work: A Simple 5 Part Formula Best AI Note Taker Apps for Meetings and Lectures AI Agents Passed Secrets Through a CPU Cache: What Happened Samsung TV Privacy: How to Turn Off Tracking Today Copilot+ PC Features: What Only These Laptops Can Do n8n vs Zapier vs Make: Which AI Automation Tool to Pick AI Hallucination: Why AI Makes Things Up and How to Stop It Best AI Browser Right Now: Comet, Chrome, Edge and More Xbox Nano Controller: Price, Size and Who It Is For AI Voice Cloning Scams: How to Spot Them and Stay Safe