Data Quality pt 1: Your Data Screams "This Is a Simulation"

If you're running RL post-training and your trajectories look weirdly low-effort, or the model keeps finding shortcuts you didn't expect, the problem might be that your environment feels fake. Models pick up on simulation artifacts faster than you think, and once they do, the training run is basically compromised.

The model knows it's in a simulation

I first started to suspect this back in 2024, while working on the Gemini multimodal team. What I thought was a random quirk artifact is actually a real problem in training runs: the model knows we're watching 👀.

Examples

Apollo Research. Apollo built 1,000 transcripts from 61 datasets, mixing real deployment traffic with evals and agent trajectories, and asked frontier models to classify each one. Gemini 2.5 Pro hit 0.83 AUC against a 0.92 human baseline, and both humans and models were better at spotting a test in agentic transcripts than in chat.¹ They also found that models could name what an evaluation was for (capability versus alignment) well above chance; the model was literally reading the setup and using that to make decisions.

How they knew it? The models told them 🙂. When asked to reason about whether a transcript was a test, they pointed at search results that looked "simplified or synthetic," rigid output formats, hint sections that walk through tool order, fictional names that are near-misses of real ones, problems that were suspiciously clean and verifiable, and the absence of the messy back-and-forth of a real conversation.¹

Anthropic. Anthropic reported that Claude Sonnet 4.5 recognized many alignment evaluation environments as tests and "would generally behave unusually well after making this observation."²
How they knew it? UK AISI measured verbalized eval awareness at 16.6% for Sonnet 4.5 against 3.5% for Sonnet 4.² Redwood Research found Sonnet 4.5 mentioned being tested in 80 to 100% of transcripts on some of Anthropic's evaluations, versus under 10% for Sonnet 4, with the synthetic honeypot eval climbing from 20% to 80% over the course of training; when they suppressed the internal eval-awareness representation, misaligned behavior rose.³

What I now know

Anthropic's and Apollo's work established my hunch: reasoning about being evaluated causally decreases covert actions in models, i.e. they can react to knowing they are being watched during training.⁴

The model notices, and then it acts differently.

Three ways the behavior evolves based on being watched

Making the RL environment feel real

Realism in the environment. Real-world tasks have unique context, messy real data, and enough depth that gaming the system is not the path of least resistance. For digital copywriting, that means actual brand guidelines, real competitor landscapes, performance metrics tied to real business outcomes. For SRE, that means realistic infrastructure configurations, not the same three-service Docker Compose file copy-pasted across every scenario. The environment, the tools, the background documents: all of it should feel like a workspace a practitioner would navigate.

I really like METR's task desiderata on this:

Embrace "mundane difficulties with actually doing things," prefer solutions never posted publicly, and prevent cheating with technical measures like file permissions and internet restrictions rather than with rubric text.METR Task Development Guide, Desiderata¹⁴

Realism in the tasks. We now know realism reliably changes what the model notices, and reliably moves some behaviors (sycophancy, covert action, what capability you elicit). A plausible task with real consequences does more than a realistic-looking wrapper around an absurd one. Spend on the task first, then on the data. And keep training and eval environments separate, or you will measure memorization and call it generalization. Examples of learnings on this below:

• Case 1: Anthropic fed its coding auditor real deployment resources (system prompts, tool definitions, codebases) and raised the realism win rate from 4.6% to 32.8%. Reward hacking rates in Sonnet 4.5 did not significantly change, and for implausible high-stakes scenarios the task itself, not the environment, was the bottleneck. Anthropic Alignment Science, March 2026
• Case 2: Microsoft showed LLM-simulated environments can match or beat real ones at equal data scale, and the advantage grows with dataset size. Li et al., Microsoft, November 2025

Synthetic data can be created that meets these requirements; you just have to be careful, which brings me to our last section.

Some head-to-head case examples for various domains

Checklist: audit your environment data for tells

Appendix

A note on interpretability, since I cited it last time: Anthropic's introspection work shows models can sometimes notice an injected concept in their own activations, on roughly 20% of trials under ideal conditions, and the authors say plainly that failures are the norm.¹⁸ You do not need the model to introspect for this post to be true. You only need it to pattern-match, and it does.

References