Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions
This paper addresses distributional shift in offline reinforcement learning by vetting synthetic transitions before policy training, narrowing the gap between training data and real-world deployment. The work demonstrates a principled approach to robustness testing under data constraints - a challenge common in regulated domains where live experimentation is limited. It reflects the same methodology Fact AI Lab applies to AI systems that must generalize reliably beyond their training distribution.
Offline reinforcement learning enables policy learning from historical datasets without live environment interaction - a critical capability when live experimentation is costly, risky, or prohibited by regulation.
The Core Problem
Standard offline RL methods suffer from distributional shift: the policy encounters state-action pairs during deployment that were underrepresented or absent in training data. This causes overoptimistic value estimates and brittle behavior. Synthetic data augmentation can fill coverage gaps, but naively adding synthetic transitions often makes things worse by introducing out-of-distribution hallucinations.
Approach
We propose a vetting pipeline that filters synthetic transitions before they enter the training buffer. The vetting mechanism evaluates each synthetic transition against three criteria:
Plausibility: Is the predicted next state consistent with the transition dynamics estimated from real data? Coverage need: Does this transition fill a genuine gap in the training distribution, or does it duplicate already-covered regions? Stability: Does adding this transition destabilize the value function estimates for nearby real transitions?
Only transitions that pass all three checks are added to the augmented training buffer.
Evaluation
We evaluate on standard offline RL benchmarks (D4RL MuJoCo tasks) and a domain-specific dataset from a maritime route optimization problem. The vetting pipeline improves final policy performance and - more importantly - reduces variance across evaluation seeds, indicating more reliable generalization.
The variance reduction result is central to the practical contribution: in regulated domains, predictability and consistency matter as much as peak performance.