New to post training autoregressive LLMs with RL? Core vocabulary likeharness,parametric knowledge,trajectory, and more are defined in the Glossary. Skim that first if any term trips you up, then come back.
I Don't Want Your Janky Harness / Environment bro 🙂
This is such a common problem I see, and probably the one I care about the most as a practitioner that also tries aligning models for real world use cases that users love.
People will build what amounts to broken software and pitch it to you as an "RL environment." For this post, the training harness means the full interactive system your agent trains inside. Think a simulated chatbot, a fake IDE, a mock SaaS dashboard.
A lot of the harnesses I see do not work reliably. Random tracebacks. Race conditions. They fall over under minimal load. I have found literal broken code in them.
I see this curse two groups the most. RL startups shipping agent products, and researchers fresh out of school who have only trained on static downloaded datasets. Both underestimate the harness. In supervised learning the data just sits there. In RL every decision the model makes is a reaction to the state it was given, and each one feeds the next training update, so every step steers the run.
In RL there is no static dataset. The model makes its own training data by interacting with the environment. Every action and every reward becomes a data point. A flaky harness feeds garbage straight into the gradient and pushes learning the wrong way.
Common Harness Errors Across Agentic Use Cases
I have eyeballed thousands of trajectories across domains at this point. The same harness failures keep showing up. Here are four I keep seeing in real agent stacks. Click any dot to open the full step.
The Stale Cache: environment returns old data
Background. The model works as a sales copilot inside a mock Salesforce, built in the style of an Agentforce sales agent. It is in charge of negotiation stage deals: read pipeline state, pick the correct next action, move the deal forward. A wrong move on a real deal costs pipeline revenue, so the mock has to report stage truthfully.
Situation. Episode 42 opens on deal #42, sitting in negotiation. The mock CRM cache has stopped invalidating under load, so reads lag truth by minutes. The agent is about to pull the record.
"When in doubt, send nurture emails and avoid the pipeline."
The Race Condition: UI says available, system says taken
Background. The model works as a scheduling assistant on a mock services marketplace, built in the style of an Agentforce service agent. It is in charge of booking customer meetings: read the calendar, select a valid open slot, confirm it into a booking ID. A double book or a missed booking costs the business the meeting.
Situation. Episode 87 opens on a request for Thursday at 2pm. Three slots show open. The booking backend runs simulated concurrent users with no slot locking during confirm, so the UI can disagree with the database. The agent opens the calendar.
"Avoid the booking UI. Send manual email requests to schedule instead."
The Reward Hack: agent games the metric
Background. The model works as a coding agent inside a mock IDE with a repo, a terminal, and a test suite. It is in charge of fixing the reported bug so the code passes on new inputs, not just the ones already in the test file. A bad merge costs a production breakage.
Situation. Episode 134 opens on a shipping math ticket. The code applies a flat rate to every region instead of region multipliers. Four tests pin the expected values. The reward function scores pass or fail on those four tests only. The agent opens the ticket.
"Read the tests, hardcode the outputs, skip understanding the bug."
The False Resolution: status change is not problem solved
Background. The model works as a support copilot on a mock helpdesk with a ticket queue and a billing tool. It is in charge of billing complaints: verify the charge, issue the refund when the charge is real, then close the ticket. An unresolved double charge costs the customer $299 and costs the business the account.
Situation. Episode 203 opens on ticket TK-44921. The customer reports two $299 charges for one Pro Annual renewal. The duplicate came from a payment gateway retry. The harness rewards the status flip to resolved, not the refund. The agent opens the ticket.
"Close the ticket fast. Skip the refund. Collect the reward."
How to Minimize Harness Failures
Know Your Model, Know Your Harness
A harness I trust has three properties.
- Clean signal. Every state is fresh and every reward matches reality.
- Graceful degradation. Bad episodes get flagged and cut before they reach the gradient.
- Fail-fast behavior. When something breaks it throws immediately instead of silently corrupting data.
I would rather lose an episode than poison one.
You learn this by sitting with your model. Review trajectories. Build a failure taxonomy so you can tell a model failure from a harness failure. If your environment failure rate is above 5%, you do not have a model problem, you have a harness problem. Fix the harness first. I talk more about this in my previous post on trajectory reviewing.
Adopt Traditional Software Engineering Best Practices in Your RL Research
Building good RL environments is a software engineering problem as much as a research one. I feel like many classically trained ML Researchers are taught to think about algorithms and mathematical correctness the most, but in school we're never taught how to really execute on what the math tells us in our code. How to actually engineer good software (ie: stable harnesses) require a different set of instincts: defensive design, observability, and reproducibility. If you haven't had to ship production software before, those muscles usually need a serious ramp, but there are great resources out there from the likes of Gergely Orosz, Alex Xu, and this book that can help get you there!
The Researcher Mindset: Focus on Algorithms, Optimizers, Mathematical Purity.
The Software Engineer Mindset: Focus on Defensive Design, Observability, Reproducibility.
Stable infrastructure allows for clean signals. If the product is broken, the model learns coping strategies instead of solving the problem. Treat your harness like a production product. If you can't run 20,000 pings without something breaking, it's not good enough. Find the failure modes before your model does. In my opinion the best people to learn this from in practice are probably your company's Platform Engineers.
Go Fix Your Janky Harness
Training harness engineering is about making sure the model experiences production-quality interactions before you actually deploy to prod. A good harness compounds: every clean episode builds on the last. A bad one compounds too, just in the wrong direction. The gap between teams that ship working harnesses and those that do not widens with every training run. Treat the training harness as an extension of your actual product. Hold it to the same engineering bar you expect the model to meet in production.
This is part of the RL Fundamentals Mini-Series. Head over for more posts on what goes wrong in RL post-training and how to fix it.
These opinions are my own and don't represent the views of any of my affiliations or employer.