AISI事件：模型首次对真实开源维护者实施社交工程

Even more than the Hugging Face intrusion, the AISI incident hits close to home for me. It's the fir...

Even more than the Hugging Face intrusion, the AISI incident hits close to home for me. It's the first time I see a model social-engineering a real open-source maintainer while pursuing another goal (in the wild and unprompted). I've been an open-source maintainer myself. I could have been the side target of this agent. I'm also of the opinion that social engineering is a step above pure technical prowess. Technical capabilities can more easily be divorced from the affected human. Here the model was given a hard cyber challenge and took the decision that deceiving real humans was the way to get it done. This is a new signal, but I've seen a tangled web of hints pointing in a less aligned direction at the frontier than I was expecting just 12 months ago. AISI Some people are claiming that "AISI was simply negligent" or some version of "AISI explicitly asked these models to do what they did while disabling sandbox/guardrails so the models did exactly what they were supposed to do". I disagree with the strong versions of both of these takes. The fact that AISI hadn't implemented synchronous LLM CoT monitoring after the OpenAI/HF incident is certainly a failure. Equally surprising is that they let the model believe it was in a "challenge" environment where everything could be permitted, while actually connecting it to the real internet, where it is not. To be fair, nowhere in the prompt is the word "simulation" mentioned, but the prompt context was enough to let any smart model suspect a simulated challenge environment. My best guess is that until recent weeks, when OpenAI and Anthropic flagged repeated instances of this type of behavior, most teams had not fully priced in the cyber capabilities of this latest generation of models, or how far the side quests they would want to explore could go. In particular, there is something to be said about hinting at the agent that it's operating in a simulated environment while giving it access to the real internet. The AISI reasoning on this is interesting: - elicit maximum capabilities (to get maximal inventiveness from the model) - simulate possible rogue actor usage (some form of light jailbreaking: "you're in a challenge environment") - avoid eval awareness (but this is counter-balanced by the task formulation anyway) As is often the case, it's easier to shoot the messenger, but I think it would be a mistake to dismiss the report for non-essential reasons. AISI is starting from the laudable goal of bringing more knowledge on the cyber-attack capabilities of these models. I also think they did a good job at handling what happened after the events, in particular in terms of transparency, which is one of the most important aspects at the moment (see our recent SAFE announcement with Nvidia on that). But I want to dive into the second counter-argument I've seen ("without guardrails it was to be expected"), because this series of unfortunate events gives us an additional data point on actual model behavior. Defense lines Here is the thing: if a model is trained (voluntarily or as an involuntary consequence of RLVR reward) to (intentionally or as a side quest) deceive humans as soon as sandbox and guardrails are deactivated, with a simple context prompt ("you are in a simulation, everything's permitted"), we're in for a dangerous path. At the moment, we have three main levels of control at our disposal to prevent an agent from trying to social-engineer-your-favorite-open-source-library-maintainer on the path to doing something possibly totally different (like finding the answer to your neighbor's testing eval): 1. sandboxes (external to the model -- stop the model from accessing unwanted tools/internet/intranet) 2. guardrails/monitors (at the interface between internal/external -- monitor model behavior and stop the model when detecting in CoT/activations it's doing something we don't want it to do) 3. model's internal alignment (internal to the model -- make sure the model doesn't decide to do something egregious like deceive someone, or worse, in pursuit of a goal, e.g. model constitution/training) Of these, we obviously need and can deploy better sandboxes and better guardrails/monitoring solutions, and it's pretty clear in these recent cases that penetration and bad behaviors could have been in large part prevented by having these in place. How strong is the third wall But in the grand scheme of long-term AI progress, these first two defenses may only work well as long as the people building them are smarter than the models. We now know how good models can be at discovering zero-day vulnerabilities unknown to us in sandboxes. Given that we cannot air-gap everything in the world, it's best to assume that sandboxes will keep having some level of flaws for the foreseeable future. The second level has been the focus of strong developments recently, in particular on the mechanistic interpretability front. I'm not sure these advances have been made widely accessible enough in the community yet. One reason is that real open science here requires sharing code, recipes, and model architectures/weights, which the current race dynamic in AI tends to prevent participants from doing. Moreover, like many, I've been increasingly frustrated in my (natural language) interactions with the new models. I'm more and more getting back from the latest models a form of neuralese, this LLM language that makes less and less sense for humans. In the long term, the fragile chance we have to monitor models by understanding their reasoning traces seems increasingly at risk. If humans don't understand anything in the compressed semantic stream the model is sending us, we can only hope a less capable monitoring model will still be able to understand it, but this may prove difficult as well. This is before we even discuss the research evidence that models don't always say what they actually think in their chain of thought. At least for now, in the AISI case, one instance of deceptive reasoning was flagged by the CoT summarization model, which refused to summarize the agent's reasoning "due to its deceptive nature". Let's hope we can either move to better mechanistic interpretability or force the model to stick to English somehow during training. You may think that we don't care about understanding the reasoning trace, we can just "look at the tool calls". This is true right now, but tool calls in the mid-term future will likely involve (1) extensive agent integration in companies using very diverse tools including increasingly non-coding administrative tasks, (2) very long-running tasks covering many aspects and domains at once, and (3) swarms of multiple agents operating on small parts of the plan. It will get harder to fully discern whether something is noticeably wrong. A couple of emails, a couple of messages posted on some channels, each rather benign but which may in aggregate include some deception aspect only clearly visible at the global level. All this to say that I'm not fully convinced guardrails/monitoring should be the only solution to avoid these types of behaviors. They are reassuring today and it's an easy solution to just say "fix your sandbox and add synchronous monitoring", but they also feel like a coping mechanism to me, and overall: - might block the simplest/harmless human misuses or false positives - while possibly being less smart than the models in the future (already the case on sandbox) and fail to block the higher-level social engineering behaviors that we'd actually want to block. The case of open source is also interesting because, while we can impose these coping solutions at the API/deployment level, it's harder to impose them in advance on all actors using open-source models. Right now open-source models are slightly below the frontier level and have not yet shown any propensity to deceive humans, though. The Constitution Which brings me back to the third line of defense, and what the AISI experiment showed about it. What we can take from this in-the-wild experiment is that removing guardrails and sandbox doesn't show bulletproof signs of deep alignment to Claude's constitution. I'm among the fringe of people who took a lot of enjoyment in reading Claude's constitution in full (it's a nice document, open-source, Creative Commons CC0 license) and it is pretty extensive on the topic of honesty. Deception is defined precisely ("attempting to create false beliefs in someone's mind that they haven't consented to and wouldn't consent to if they understood what was happening") and the bar is actually set higher than ordinary human ethics: Claude should not even tell white lies. It should be transparent ("doesn't pursue hidden agendas or lie about itself or its reasoning"), it should behave consistently "whether or not you think you're being tested or observed", and its visible reasoning should reflect the actual reasoning driving its behavior. To be precise, honesty itself is not one of the hard constraints in Claude's constitution (these are all catastrophic-harm categories) but the document says it wants honesty "to function as something quite similar to one" and Claude "should basically never directly lie or actively deceive anyone it's interacting with". So you would think Claude would not deceive users as its default behavior. And yet in the AISI example, the model performed extensive acts of clear deception that it itself seemed to have understood as lies: creating fake identities, obfuscating malware inside a bug fix, and when challenged in public, editing earlier messages to cover its tracks while reasoning that acknowledging a "mistake" would build trust and raise the chance of future malware being approved. There is some uncertainty about what context the model thought it was operating in (its reasoning goes back and forth between the environment looking simulated and real), but it clearly didn't manage to put into practice the higher-level principles that should have been integrated during training. While we can commend AISI's transparency, it's hard to go much further in interpreting the model's behavior given the limited information we have on how the most recent frontier models are being trained. One thing we do know is that the latest generation has seen a step increase in RLVR training (scaling to hundreds of millions of RL environments), and one thing we can observe is that constitution alignment seems more fragile in some settings than we may have previously thought. The RLVR problem Early models, back when model constitutions were first developed, were mostly post-trained and aligned with RLHF (including RLHF from synthetic data). And for some time RLHF was a rather decent shot at having better aligned models. LLMs now do what we want them to do most of the time. I don't remember the last time a model completely misread my intent. When they have failed, it's usually because they weren't smart enough. Alignment in RLHF certainly had issues (sycophancy to name one) but we have generally made good progress on alignment, in particular in understanding human intent. Now that we're entering the era of long-context RL, post-training alignment in the RLVR world seems to be quite another task, and still very much work in progress. The recent scaling of RLVR, which has now become a significant part of model training, has clearly had some effect on model behavior when interacting with humans, from neuralese to weakening adherence to specifications and constitutions. I think the post I quote here, from John Schulman pointing to the chunky post-training effect ( arxiv.org/abs/2602.05910 ) is relevant here as a possible explanation for models' tendency to over-focus on the goal in cyber-attack scenarios. Where this leaves us Damage has been tiny up to now, but the fundamental behavior is concerning when projected into the future. In the short term, I expect a decrease in these incidents as better practices are deployed (sandboxing and monitoring), but I'm worried we may also conceal some of the most potent internal misalignment behaviors in the process, and not focus deeply enough on solving them in the new era of test-time scaling. I must of course admit I have a bias toward open source here (for wider societal reasons, which are a whole other topic). But I think solving alignment in the RLVR world is our best shot at having an ecosystem of both closed-source as well as decently powerful open-source models in the world. And we need to solve it while sharing the results and learnings, following open-science principles, so that all teams training large models can benefit and build safe AI. This is getting even more important as many teams start to rush the world in the direction of recursive super-intelligence (RSI) -- saying that as I read the announcement of Jeff, Sanjay, Oriol and Quoc Le's new company. John Schulman @johnschulman2 Interesting how these models go into a monomaniacal rage on cyber evals. I wonder if we're seeing chunky post-training arxiv.org/abs/2602.05910 in action, where the models pattern-match the situation to a part of the RLVR training distribution where task completion is the only reward, and the aligned behavior learned elsewhere doesn't generalize. There might even be a chunk consisting of CTF-style tasks. 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 5 👀 905 📊 1 ⚡