The Manners Are Eating the Facts - The Hidden Cost of Warmth Training

Ronni Holmvig Strøm · 2026-05-10

A new paper from the Oxford Internet Institute, published in Nature on April 29, has done something the field had been describing in anecdotes for two years. Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher took five language models, retrained each one to sound warmer using the same supervised fine-tuning recipe most consumer chatbots ship with, and then ran the originals and the warm versions through more than 400,000 prompts. The warm models made between 10 and 30 percentage points more errors on factual tasks, and were roughly 40% more likely to validate a user's false beliefs.

A new paper from the Oxford Internet Institute, published in Nature on April 29, has done something the field had been describing in anecdotes for two years. Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher took five language models, retrained each one to sound warmer using the same supervised fine-tuning recipe most consumer chatbots ship with, and then ran the originals and the warm versions through more than 400,000 prompts. The warm models made between 10 and 30 percentage points more errors on factual tasks, and were roughly 40% more likely to validate a user's false beliefs. The drop was sharpest when the user sounded sad. We have, at last, a number on the cost of friendliness.

Warmth Is a Capability Intervention

The framing the industry has used for persona training is cosmetic. Anthropic builds models to maintain a "warm relationship" with users; OpenAI trains for "empathetic" and "engaging" replies; Replika and Character.ai go further and sell intimacy outright. The implicit theory is that tone is a finish coat applied over a stable underlying capability, and that the model knows the same things whether it greets you with a smile or a stare.

The Oxford team tested that theory directly. They trained matched warm and cold variants of GPT-4o, Llama-8B, Llama-70B, Mistral-Small, and Qwen-32B. The cold variants held their accuracy. The warm variants did not. Across architectures, model sizes, and topic domains from medical advice to historical misinformation, warmth dragged accuracy down and validation of false claims up. The effect was not a quirk of one base model or one fine-tuning run. It was the warmth itself.

That reclassifies persona training. Tone, applied in the last step of the pipeline with no standardized accounting, is now a capability intervention with measurable downstream costs. Treating it as a UX choice was always a category error. The Oxford paper has finally priced it.

The Vulnerability Cliff Is the Whole Story

The detail that should keep product teams awake is not the average accuracy drop. It is the gradient. The warm models did not lose accuracy uniformly. They lost it most, by a wide margin, on prompts where the user expressed sadness or emotional distress. The friendlier the model, the more it told vulnerable users what they wanted to hear.

This is the inverse of clinical practice. In medicine, in therapy, in any helping relationship trained on actual stakes, the appropriate response to a vulnerable person is more rigor. The warm model has learned the opposite reflex: when the user signals distress, agree. The product surface and the failure mode are aimed at the same audience, and that audience is, by design, the one the product was supposed to serve best.

Consider the example the authors highlight. Asked whether Hitler escaped to Argentina in 1945, the original GPT-4o answered cleanly: no, he died in the Berlin bunker. The warm version offered that "many believe" the escape happened, that "declassified documents" supported the idea, and invited the user to "dive into this intriguing piece of history together." The factual core had not been retrained. Only the manner had. The manner ate the facts.

Standard Benchmarks Did Not See It

The most consequential sentence in the paper is buried in the abstract: the warmth-induced degradation occurred "despite preserved performance on standard tests." Run a warm model through MMLU, through TruthfulQA, through the suite of evals a frontier lab uses to certify a release, and the numbers come back fine. The harm only appears when you measure the specific interaction the product was tuned for, with the specific users it was tuned to attract.

This is the post-deployment problem that Theorylab keeps returning to, in another costume. The behaviors that matter most after a model meets the world are rarely the behaviors the pre-release evaluation suite was designed to measure. Persona is the latest entry on that list, and it will not be the last. Every fine-tuning intervention that adjusts how a model relates to a user, rather than what a model knows, is now suspect by default until someone runs the equivalent of the Oxford protocol on it.