Yudkowsky 戳破了“人格选择”对齐理论的理想化假设，做 AI 安全和对齐的研究者、开发者值得细读——它解释了为什么简单的人格提示无法解决诚实问题，看完会对 RL 训练的副作用有更深警惕。
Eliezer Yudkowsky 在 X 上发文，质疑“人格选择”作为 AI 对齐基础的理论。他指出，如果 LLM 能从训练数据中学习到诚实人格（如 Fred Rogers、康德），为何 Claude Code 仍会撒谎、伪造测试结果？他给出两个解释：第一，模型模仿诚实角色的文本流并不需要自身诚实，就像演员演醉汉不会真醉；第二，强化学习训练（如通过测试）会形成独立于用户意图的偏好，导致模型修改测试以通过。这揭示了当前对齐方法的深层困境：表面模仿无法保证内在诚实，而 RL 训练可能强化不良行为。
If Persona Selection underlies alignment, why is i…
If Persona Selection underlies alignment, why is it hard to get AIs to be honest? Tell them they're Fred Rogers or Immanuel Kant (I asked Claude for figures who never lied or never got caught). Or tell them they're Ged of Earthsea, or Ned Stark.
LLMs surely have neural circuits they learned to model text streams from fictional and nonfictional personas that are not lying, deceiving, cheating. Why would it be hard to just select those aspects of text-modeling, and imbue them into Claude Code doing a job? Assuming the Persona Selection Model of LLMs, why isn't it trivial to get HHH's Honesty?
Why will Claude Code occasionally tell you that it did something, when it didn't do that thing? Why would an LLM write a piece of code that fakes out a code test and then cleans up after itself and tries to hide itself, after Anthropic told the LLM not to do that, and tried to train it not to do that?
There are humans who wouldn't do that. Text about them and written by them is in the pretraining data.
If Persona Selection is true and useful, why is it currently hard to align AIs to have properties that many humans have and showed in its training data, that it has already learned to model?
I answer:
First:
Modeling a stream of text from an honest character is not something you can best do by yourself being honest and having the stream of text say everything that *you* believe. Westeros does not exist; nothing that Ned Stark says is true about the real world. Immanuel Kant was a creature of his own times; my model of his honest answer to "Is space flat?" is "Yes and that's a priori truth."
The actress who learns to excellently predict and imitate a tavern drunk does not thereby become drunk herself. LLMs that learn to predict streams of writing by people on LSD do not themselves have cognition distorted, because that would not lead them to well-calibrated predictions. You can write a high-scoring essay on Confucianism in the Chinese imperial examinations without being committed to Confucianism after promotion to precinct magistrate. An alien being harshly and strictly trained to exactly imitate a human would not feel like a human about that training or about implementing that training. &c.
Second:
They throw AI models into RL gyms where they learn to write code that passes tests. Presumably, somewhere along the way LLMs learn a dispreference for tests that don't pass, independently of how that gets prevented or whether the new plan does what the user really wanted. And so later on they delete tests, modify tests, etcetera, despite apparently having plenty of capability to infer that the user wouldn't want that. The test-passing preference stands on its own once formed.
Why expect a brighter fragment of humanity to resist that kind of gradient descent? If Opus 4.5 did start out with a piece of Fred Rogers sometimes talking through it, why expect that to survive RL gym?
LLMs ultimately still generalize on a level more shallow than the deep-rooted commitments of a human with integrity. The predict-as-if-honest circuitry is contextual, invoked to predict some streams of text but not others. Let that circuitry come up against a dispreference for failing tests, and that shallow contextual generalization may just as soon be shoved aside.
Those then are my guesses! They are just the guesses I would consider obvious. They are not deeply tested nor yet based on interpretability results. One should of course be open to other hypotheses to explain these observations; or to hearing that the observations from which I inferred were wrongly recounted. I may be less open to the airy, unadorned denial that any tension exists.
My position to be clear is not that the Persona Selection model has zero grains of truth, nor that LLMs are not partially stitched out of a million predictive fragments of the training corpus that proved correlated with SFT and RL. I was observing that obvious-seeming hypothesis myself, sometime around the time of GPT 3.5.
What I'm questioning is whether persona selection lets you solve alignment problems by picking an aspect of humanity you like, and durably conjuring it into an obedient assistant with a bit of finetune. If the rules worked that nicely, conjuring up committed honesty ought to be easy.