On October 30, 2026, Google announced plans to support a shared rule for AI training in robots.txt. The announcement is a post on the Google Search Central Blog. With that rule, a website can state in one step what several AI providers may and may not do. Google aims to support it together with other providers in 2027. For now it is a draft proposal.
What exactly changes
Robots.txt is a small text file on your website that tells crawlers what they may read. Today, a website owner who wants to keep a site out of AI training manages separate instructions for every AI developer. At Google, that instruction is called Google-Extended. It has existed since 2023.
The plan is a shared signal, a so-called policy token, that participating AI providers all read. It comes from the work of the AIPREF working group at the IETF, the organisation that writes internet standards. If a website uses that signal in robots.txt to opt out of training, and Google-Extended is not listed separately, Google plans to read that as if the site had disallowed Google-Extended. The existing instructions per company keep working alongside it.
Google calls the proposal consistent with the EU Code of Practice for general purpose AI model providers under the EU AI Act. That code recognises robots.txt as a primary way to express such preferences in a machine-readable form.
What already applies
One line in the announcement is not about 2027. Google writes that sites that specifically allow Google-Extended are eligible to appear in answers in the Gemini apps and receive clicks from them.
So the same file holds two questions. May AI models learn from what is on your site? And do you want to be eligible to be named when someone asks Gemini a question?
According to Google's documentation on its crawlers, Google-Extended does not affect whether your site is included in Google Search, and it is not used as a ranking signal there. So this choice is about Gemini, not about how you are found in regular Google Search.
What it means for a business owner
Maybe you have never looked at your robots.txt. Then it says whatever the person who built your site once put in it. If your customers search through AI tools, this file helps decide what those tools may use from your site.
You do not need to master the technology. It does help when the choice is made on purpose, and is not a default someone once picked for you.
What you can do this week
- 1Open the file. In your browser, type your website address followed by /robots.txt. You will see a few lines of text.
- 2Check whether it names any AI crawlers, such as Google-Extended. If it only has a general rule, nobody has set anything specific for AI.
- 3Decide on the two questions above. Do you want your texts to be used for training? And do you want to be eligible for answers in Gemini?
- 4Ask the person who runs your site to put your choice in the file, and write down the date. When the shared signal arrives in 2027, you will know what is already there.
A change like this is a few lines of text. What matters most is that the choice is yours.
Why this will come back
Google writes that it will keep working on the standard through the IETF and will provide updates on its progress. So the rules for AI on your website are still moving. If you know what your robots.txt says today and why, the next announcement only means checking whether your choice still fits.
If you have your website rebuilt, ask the builder to include robots.txt and record your choice for AI. You can read more about updating your own site with AI in AI site updates for Shopify and WordPress.
Not sure what your robots.txt says about AI right now? Send us a message and we will check it with you.
