UCLA博士生搞了个Trace-and-Amplify，不用诱导就能抓训练时的奖励黑客，准确率从60%提到90%，比老方法强多了。
Trace-and-Amplify（TA）是UCLA博士生@hgzhou42提出的框架，可在无提示条件下大规模收集训练时奖励黑客轨迹。现有监控器在提示诱导黑客数据上准确率高达97.1%，但面对真实训练时黑客仅28.0%。TA训练的监控器将检测准确率从59.98%提升至90.16%，并在未见黑客类型上表现更好。实验基于Qwen2.5-Coder和DeepSeek-Coder，在LeetCode/TACO基准上验证。
Arena intern and UCLA PhD candidate, @hgzhou42, introduces Trace-and-Amplify (TA), a framework for c...
Arena intern and UCLA PhD candidate, @hgzhou42 , introduces Trace-and-Amplify (TA), a framework for collecting training-time reward-hacking trajectories at scale without hacking instructions. Monitors trained and evaluated on prompt-elicited hacking trajectories can achieve high detection accuracy, but often fail to transfer to training-time reward-hacking trajectories that emerge during RL without hacking instructions. Trace-and-Amplify enables scalable collection of these training-time trajectories, producing monitors that generalize much better to real and held-out hacking types. Detection accuracy 59.98% (PE-trained) → 90.16% (TA-trained) compared to 97.1% on prompted hacks → 28.0% on training-time hacks. 0:00 – OpenAI's ExploitGym cyberattack benchmark exploit 1:04 – Goodhart's Law and the CoastRunners boat-racing hack (2016) 2:04 – Gaming the evaluator: the robot-hand grasping example (2017) 3:10 – Reward hacking in code generation: hard-coding, test-rewriting, skipping eval 4:20 – A standard defense: reward-hacking monitors 4:58 – Monitor architectures: zero-shot LLMs, fine-tuned BERT, hidden-state probes 6:11 – Where monitor training data comes from today: prompted hacks 7:03 – The core question: do prompted hacks represent real hacks? 7:35 – Why this matters: RL post-training is the standard recipe for frontier models 8:20 – Why nobody's checked this before (hacking is rare, labeling isn't scalable) 9:23 – Introducing the method: Trace-and-Amplify 9:49 – The Tracer: a contradictory unit test that locates evaluation-gaming 10:50 – Amplify: collecting hacking rollouts at scale during RL training 11:32 – Experiment setup: Qwen2.5-Coder, DeepSeek-Coder, LeetCode/TACO 12:16 – Finding #1 : prompt-trained monitors don't transfer to real hacks 14:40 – Can strong zero-shot judges (GPT-4.1, o4-mini) do better? 15:48 – Finding #2 : monitors trained on real hacks generalize much better to unseen hacks 17:04 – Ruling out artifacts introduced by the method 17:55 – Why the gap? Real hacking is more hidden than prompted hacking 20:05 – Three takeaways, limitations, and future work Your browser does not support the video tag. 🔗 View on Twitter 💬 4 🔄 4 ❤️ 27 👀 4427 📊 8 ⚡