Lifestyle
OpenAI, Google, Microsoft and Anthropic Say They Will Adopt One Signal for Opting Out of AI Scraping
The Verge reports that four AI companies say they will adopt a shared signal placed in a site's robots.txt file. Site owners and publishers would no longer need to tell each company separately whether their content may be scraped to train AI.
About 4 min read
What happened
According to The Verge on October 30, 2026, OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that lets publishers opt out of AI scraping, meaning AI companies' automated programs (crawlers) would not collect a site's content. The Verge notes that Search Engine Roundtable reported the news earlier.
The Verge says publishers can use it to apply their AI training preferences, that is, whether content may be used to train AI models, to several AI companies at once rather than setting them for each one. Currently, sites must state such preferences to each AI company separately.
How the "policy token" works
The "policy token" can be understood as a shared marker a website leaves for AI crawlers. The Verge reports that the signal was proposed at the internet standards body the Internet Engineering Task Force (IETF) by Kevin Kelley, a technical lead for standards at Anthropic.
As The Verge explains, the signal goes inside a site's robots.txt file. This file has long been used to indicate which web crawlers may or may not scrape a site, whether for search indexing, archiving or AI training.
The Verge says OpenAI, Google, Microsoft and Anthropic all state that, when they encounter this shared signal, they will restrict their AI crawlers' access to that site.
Company timelines and differences
| Company | Status or approach as reported by The Verge |
|---|---|
| Anthropic | Has adopted the signal |
| Plans to support it next year | |
| Microsoft | Plans to support it next year; will also offer finer-grained controls to opt out of AI training or AI grounding (pointing an AI model to a verified source of information) |
| OpenAI | Says it will implement it in the "coming months" |
The Verge notes that Microsoft will offer finer-grained controls that let publishers opt out of AI training or AI grounding. The Verge describes AI grounding as pointing an AI model to a verified source of information to reduce "hallucinations" (AI producing false content). Microsoft also says sites will still appear in Bing search results regardless of their preferences.
What it means for site owners
The Verge notes that publishers face pressure from AI crawlers that keep scraping large amounts of site content for use in AI models. As described, the new signal lets site administrators apply their AI training preferences to several companies at once, without setting them for each one separately.
The Verge also mentions other approaches against unauthorized scraping, such as the Really Simple Licensing (RSL) standard, which lets publishers charge visiting crawlers, and Cloudflare, which blocks AI crawlers by default. The new signal is a different option: the site states its preference, and the AI companies say they will comply.
FAQ
What is this new standard?
According to The Verge, it is a shared "policy token" signal placed in robots.txt, which publishers can use to state their AI training preferences to several AI companies at once.
Who proposed the signal?
The Verge says it was proposed at the IETF by Kevin Kelley, a technical lead for standards at Anthropic.
Which companies already support it?
According to The Verge, Anthropic has adopted it; Google and Microsoft plan to support it next year; OpenAI says it will implement it in the "coming months."
After opting out, will a site still appear in search results?
The Verge reports that Microsoft says sites will still appear in Bing search results regardless of their preferences. The report does not explain other companies' approaches.
How does Microsoft's approach differ?
According to The Verge, Microsoft will offer finer-grained controls that let publishers opt out of AI training or AI grounding.
Browse the latest news in this topic
Lifestyle
AI Infrastructure Spending Accelerates: Big Four Tech Firms Plan Over $740 Billion in Capex This Year as Investors Worry About Cash Flow and Debt
TechCrunch reports Alphabet, Amazon, Meta and Microsoft plan to spend over $740 billion this year, seen as AI infrastructure outlays. Three show negative free cash flow, investors question returns, and Anthropic's IPO plans may be affected.
Lifestyle
Report: Anthropic Likely to Receive a "High Junk" Credit Rating After IPO
TechCrunch, citing the Financial Times, reports Blackstone now expects a double-B rating for Anthropic after its IPO, contrary to an earlier investment-grade claim, which could make fundraising costlier.
Lifestyle
Sam Altman Says RAM Is Better in Data Centers Than on Your Desk, as the Memory Shortage Pushes Up Device Prices
On a podcast, OpenAI CEO Sam Altman said it is "quite terrible" for the world's limited memory supply to sit in personal computers. The Verge notes that billions spent on chips by big buyers are why laptops, phones and other tech are getting more expensive for ordinary consumers.
Lifestyle
OpenAI Publishes Safety Details for Biology AI Model GPT-Rosalind-6: Rated "High" Capability, Limited to Vetted Researchers and Partners
On October 30, 2026, OpenAI published a system card addendum for GPT-Rosalind-6, an AI model for biology and drug discovery. It can support advanced research better than openly available ChatGPT models, so it is offered only to vetted scientists, institutions, companies and government partners. All details are OpenAI's own statements.