Sim-to-real transfer — why your policy fails on hardware

The moment is familiar to anyone who has trained a policy: reward curve flat at the ceiling, a hundred successful rollouts in simulation, and then the robot on the bench does something that looks like a different task entirely.

This is not random. Sim-to-real failure has a small number of recurring causes, and most of them are diagnosable before you ever touch hardware.

The gap is specific, not mysterious

“Reality gap” is a phrase that hides the actual problem. The gap always lives in one of four places:

1. Dynamics. Friction, damping, motor torque curves, backlash, mass distribution. Your simulator’s defaults are almost certainly not your robot’s values.

2. Observation. In simulation you read the true state. On hardware you read a sensor with noise, latency, drift and occasional dropouts. A policy trained on perfect observations has never learned to survive a bad one.

3. Actuation. Commanded torque is not applied torque. There is a delay, a saturation limit, and a controller in between that has its own dynamics.

4. Timing. Simulation steps deterministically. Hardware has variable loop times, dropped frames and communication jitter. A policy that implicitly learned to depend on a fixed control interval will degrade the moment that interval moves.

Before touching your training setup, determine which of the four is responsible. Treating a timing problem with more domain randomization on friction wastes weeks.

Diagnosing before you deploy

Log a real rollout and replay it in simulation. Take the exact action sequence the hardware executed, apply it in sim from the same starting state, and compare trajectories. Where they diverge tells you which dynamics are wrong. This one test is worth more than any amount of speculation.

Inject the noise you measured. Characterise your sensor — record it stationary for a minute and look at the actual distribution. Then put that noise in your simulator, not an assumed Gaussian.

Measure your real latency. Command to observed response, end to end. Put that delay in the simulation. Latency is the most commonly ignored and most commonly fatal component of the gap.

Domain randomization, done properly

Domain randomization trains across a distribution of dynamics so the policy learns something robust rather than something exploiting one exact configuration.

It works. It is also routinely misapplied.

Randomize what you are uncertain about, at the scale of your uncertainty. If you measured your robot’s mass to within 5%, randomizing mass by ±50% does not add robustness — it adds a much harder problem and a more conservative, worse policy.

Randomize the four categories, not just the obvious one. Most implementations randomize friction and mass and stop. Sensor noise, latency and control-interval jitter are usually the ones that matter more.

Check the trade-off. Excessive randomization produces a policy that is mediocre everywhere — it learns to ignore its observations because under enough noise they were never reliable. If simulation performance collapses as you widen the ranges, you have gone too far.

A reasonable schedule is to start narrow, confirm the policy learns the task at all, then widen progressively until simulation performance starts to degrade — and stop just before that point.

System identification beats guessing

Domain randomization covers uncertainty. System identification removes it, and removing it is better.

Run a set of known excitation trajectories on the hardware, record the response, and fit your simulator’s parameters to match. An afternoon of this typically narrows friction and damping estimates enough that your randomization ranges shrink dramatically — which makes the learning problem easier and the resulting policy sharper.

The general principle: identify what you can measure, randomize only what you genuinely cannot.

Design the observation space for the real robot

A large fraction of sim-to-real failure is decided before training, in the choice of what the policy observes.

• If a quantity is not reliably measurable on hardware, do not put it in the observation space, however convenient it is in simulation.
• Prefer observations that are robust to calibration drift — relative positions over absolute ones, for instance.
• Include a short history of observations. It lets the policy infer velocity and compensate for latency implicitly, and it is close to free.

Asymmetric actor-critic helps here: let the critic see privileged simulator state during training while the actor sees only what hardware will provide. You get the training signal without the dependency.

Smooth actions matter more than the reward curve suggests

Policies frequently learn high-frequency action patterns that are harmless in simulation and destructive on hardware — they excite resonances, overheat actuators, and wear out gearboxes.

Penalise action rate in the reward, or filter the output. Your reward will look slightly worse in simulation. Your hardware will survive, and the policy will usually transfer better anyway.

A sensible deployment order

1. Train in simulation with identified parameters and modest randomization
2. Validate on held-out simulated dynamics the policy never saw
3. Replay a real trajectory in simulation and confirm the divergence is small
4. Deploy with conservative action limits and a hardware kill switch
5. Log everything from the real rollouts
6. Feed real trajectories back into system identification and retrain

Step 6 is where teams that succeed differ from teams that do not. Sim-to-real is a loop, not a handoff. The first deployment is an experiment that tells you what your simulator got wrong.

The cultural point

The temptation, when a policy fails on hardware, is to train longer. It is almost never the answer. A policy that solves the simulated task has solved the simulated task; running more steps makes it better at exactly the thing that is not your problem.

The work is in making the simulation resemble the robot — which is measurement, not machine learning.

---

SimRoboX is built around this loop: scene authoring, domain randomization and the evaluation tooling that tells you honestly whether a policy is ready for hardware.