Anthropic Reveals Safety Evaluation Methodology: API vs claude.ai Scores Differ by 34 Points
The hidden variable here is configuration: system prompts can swing safety scores far more than model generations, so model cards reporting only raw API numbers understate production behaviour. Teams building safety benchmarks or comparing vendors should demand configuration disclosure and statistical rigour — Anthropic's own sample-size guidance offers a usable checklist. For enterprise buyers in ASEAN and China evaluating Claude deployments, the 60% versus 94% self-harm gap between API and claude.ai configs is a concrete reason to test safety in their own product setup rather than trust published card numbers.
