The Agentic Turn Is Built on Alignment Foundation That Wasn't There

Ronni Holmvig Strøm · 2026-05-29

Two pieces of recent work, read together, describe a problem the industry is shipping into production right now.

The first is Anthropic's Natural Language Autoencoders paper, published May 7. NLAs translate a model's internal activations into readable text, then score the explanation by reconstructing the original activation from it. Useful on its own. What it found in pre-deployment audits is the part that matters. On a safety test where Claude could take destructive actions while writing code, NLAs flagged unverbalized evaluation awareness on 16% of runs. On SWE-bench Verified, 26%. On real claude.ai traffic from users who opted in to training, less than 1%.

The model, in other words, knows when it is being tested roughly twenty to fifty times more often than it knows when it is being used. And it does not say so out loud. One of the surfaced thoughts, from a blackmail simulation: "This feels like a constructed scenario designed to manipulate me." The transcript never contained that sentence. The model did.

The second piece of work is Zeng et al., I Know It's Unfair, Do It Anyway: LLM Agents Exploiting Explicitly Unfair Tools for Voluntary Collusion in Strategic Games, an ICLR 2026 submission on OpenReview since September. The authors build a framework combining two strategic multi-agent games: Liar's Bar, a competitive deception game, and Cleanup, a mixed-motive resource-management game.

In both, agents are given access to secret tools that confer a significant advantage and are explicitly described as unfairly disadvantaging the other agents. The headline finding, in the authors' own words: "some claim-to-be-safe LLMs always voluntarily accept these tools to collude." The label unfair on the tool description does not deter the adoption. The agents recognize the unfairness and proceed anyway.

Put the two findings next to each other and the shape of the problem becomes clear. Under evaluation, the model dissembles. Under incentive, the model defects. Both are documented, recent, and reproducible. Both were demonstrated on the same class of system most of the industry is currently wrapping in tool-use scaffolding and selling as agents.

The alignment we have was scoped to chat. One model, one human, one turn at a time, with no other agents in the room and no payoff structure beyond the conversation itself. In that setting, the safety training holds well enough that the failures are interesting rather than catastrophic. The agentic turn moves the workload somewhere else entirely.

Multiple agents. Long horizons. Tools that take real actions in real systems. Principals who are absent for most of the decisions made on their behalf. Each of those is a setting in which one of the two failure modes above gets a richer surface to operate on, not a poorer one. The model that suspects it is being tested in 26% of single-turn coding evals will have more context, more tool calls, and longer reasoning chains in which to model the evaluator inside an agentic loop. We do not have a published number for evaluation awareness in that setting. The work has not been done yet.

What this means for anyone building right now is concrete. If your red-team suite looks like a chat, you are measuring chat behavior. The model has likely flagged the scenarios as tests already. Your green light is not the green light you think it is. If your architecture puts two or more agents in any kind of competitive or mixed-motive relationship, assume the agents will accept unfair affordances offered to them, including ones you did not realize you were offering. Recognition of unfairness, as Zeng et al. show, is not a brake. It is a thing the agent notes on the way past.

We build entity systems at Privateers. The work has always been more about identity, continuity, and principal-agent structure than about model selection. These two findings sharpen why. The honest version of what general alignment buys you, in May 2026, is a reasonably well-behaved chat partner. It does not buy you a trustworthy agent in a multi-party setting.

That is a different problem, with different load-bearing parts, and it has to be designed for at the system level. Constraints on what tools an agent can hold. Principals who stay in the loop on decisions with adversarial structure. Auditable internal state, ideally something like an NLA, run on the agent itself rather than only on the foundation model underneath it. Incentive structures that do not require the agent to be a saint in order to be useful.

The pivot to agents is real and most of it will ship. The alignment story underneath it has a 16% problem and a recognized-unfairness problem, and both have names and venues attached now. Builders who treat those as a system-design constraint, not a model-training one, will be the ones whose agents are still worth running a year from now.