The New robots.txt AI Training Opt-Out: What Designers and Builders Should Do With It

On Friday, Google, Microsoft and OpenAI all announced they will honor one shared line in robots.txt that tells AI companies whether they may train on your site, and Anthropic, whose Kevin Kelley authored the underlying IETF draft, says development has started there too (Search Engine Journal, Google Search Central). The line looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That is the actual example from the draft, and it is worth pausing on, because that one block is a design decision, not a sysadmin chore.

Image: Google Search Central, via Supporting a shared AI training control for websites

Why the coverage is missing the design angle

Most of the reaction frames this as publisher news. Media companies, data licensing deals, the fight between newsrooms and AI labs. That framing is not wrong, but it hides who actually ships the most scraped pages per capita: designers and builders. Portfolio sites, case studies, component libraries, tutorial pages, client launches. Every portfolio on the web is a training set, and until now the only real options were a stack of per-crawler rules with different syntaxes, each one a homework assignment, or silence.

The design angle is this: for the first time, the web has a consent mechanism for how creative work gets used by AI, and it is path-specific instead of all-or-nothing. Block everything, allow the press kit. That is the same permissions thinking designers already apply to IAM screens and privacy settings, except this time the product being protected is your own work.

My actual position

I think this control is overdue, and I think the reflexive response to it will be wrong.

The overdue part is obvious. For a decade robots.txt was a dev afterthought, a file you copied from a template and forgot. It is now the consent layer for creative work on the web. If you build or ship websites for a living, content usage belongs in the handoff document, next to the favicon and the meta tags. Treating it as anything less is a gap in the deliverable, and clients will start asking about it because their lawyers already are.

The reflexive part is where I will push back on my own industry. The moment a block button appears, a lot of creatives will mash it site-wide, and I understand the feeling. Your unpaid labor became a multibillion-dollar training corpus and nobody asked. But look closely at how the companies implemented this, because the fine print is a design decision too.

Microsoft was explicit that restrictions will "limit visibility of your brand in answers" from AI tools. Google's version folds training and grounding (being cited inside Gemini answers) together under Google-Extended, so one shared opt-out line, unless you specify otherwise, takes you out of both. In other words, the button that stops the models learning from you is often the same button that stops them citing you.

That matters because AI answers are becoming a discovery surface. When a client asks an assistant for a motion designer in Cleveland, or a founder asks which studio built a certain product page, the cited portfolio wins the work. Blocking everything trades your influence over what models learn for absence from where you get found. That is a legitimate trade for some people. It is a bad default.

So my position: use the mechanism, but use it like a designer. Separate three decisions that most coverage glues together: crawling (do you let bots in), training (do they learn from you), and grounding (do they cite you). The new line only speaks to training. Crawling rules stay exactly as they were, per the draft. Grounding is handled separately at Microsoft and bundled at Google. Choose each one on purpose.

What designers and builders should actually do

1. Add the group now, even though nobody enforces it yet. Google aims for 2027, Microsoft starts honoring new preferences on February 1, 2027, OpenAI says the next few months. Legacy parsers just ignore the line, so it is harmless day one and you will not have to remember later.
2. Decide per path, not per site. The draft's own pattern is the right one: work-in-progress and full case studies can say no, press kit and about page can say yes. Longest-match precedence means you can be surgical.
3. If you or your clients use NoArchive or NoCache, calendar two dates. Microsoft stops recognizing those tags as training or grounding preferences after March 1, 2027, and a temporary preference window in Bing Webmaster Tools opens December 15 for sites still using them. Miss that and you silently default to allow.
4. Remember this does not touch search rankings. Google says Google-Extended does not affect inclusion in Google Search. You are not choosing between AI rules and your SEO. You are choosing what AI systems may do with content they already fetched.
5. Put content usage in the launch checklist. Every site you ship this year should ship with a deliberate answer to "may AI train on this", written down, per section of the site. That answer can be yes! "Yes, and cite me" is a strategy too.

If you are shipping fast with an AI builder, this takes five minutes before launch. I build most of my quick internal tools and client prototypes on Base44 (it is the #1 tool I recommend for designers who want working apps without a dev loop), and adding a robots.txt block is exactly the kind of small, deliberate detail that separates a launched product from a demo.

The takeaway

The interesting thing about a permission system is not who blocks. It is that usage is finally a designed decision. Before Friday, the answer to "can AI train on my portfolio" was a shrug. Now it is a document you author, with per-path nuance, the same way you author a design system. The people who treat it as part of the craft will get more out of the next decade of AI than the people who treat it as a war.

So: block, allow, or curate? What is your portfolio going to say?

Related on NewForm

• r/opensource Just Banned AI Projects. Solo Builders Will Pay the Price
• Shopify Canvas: What AI Store Building Actually Means for Designers
• Shopify Canvas: When Building a Store Becomes a Conversation, What Happens to Designers?