AITOPAI热报
GlobalStandardProposedModel providersDeployers

Unified ai-training directive in robots.txt for signalling AI training data restrictions

robots.txt 新增 ai-training 指令统一声明 AI 训练数据抓取限制
IETF·Published 2026-10-31·Checked by 管理员 on 2026-11-01

Google, Microsoft, OpenAI and Anthropic agreed to jointly adopt a new ai-training directive in robots.txt, based on an IETF draft specification, allowing websites to declare that their content cannot be used for AI training. OpenAI will support it within months, Bing from 2027-02-01, and Google during 2027.

Source ↗Remind meGlobal · All

What it means for you

  • Website owners can declare AI-training crawling restrictions to all four major AI companies with a single ai-training directive in robots.txt, instead of per-company tags.
  • Model providers must update their crawlers to honour the tag per their announced timelines: OpenAI within months, Bing from 2027-02-01, Google during 2027.
  • The standard is still an IETF draft and compliance is voluntary; it carries no legal enforcement yet.

    • What to do

    ComplianceHave legal/compliance assess deploying the ai-training tag in robots.txt to protect proprietary content
    EngineeringModel providers: schedule crawler support for the directive and implement parsing logic
    LegalTrack the IETF draft's progress and each company's adoption of the final spec

    Key dates

    2026-10-31Published

    Source documents

    Source ↗

    Coverage

    Stories in the feed
    Editorial summary, not legal advice; dates and status follow the original document. · Report a correction