Robots and other embodied agents learn by acting in the physical world, but the world does not remain fixed. A controller changes as training proceeds; hardware wears; payloads vary; and friction, gravity, wind, sensor noise and contact conditions can shift. At the same time, real-world interaction is costly, so an agent cannot simply discard its past and collect a fresh on-policy dataset whenever conditions change.
This creates a fundamental problem for standard off-policy reinforcement learning. Historical transitions stored in a replay buffer may have been generated by older policies and under older physical dynamics. When those stale transitions are used as if they still described the current environment, value estimates can become biased and the policy may learn actions whose predicted outcomes are no longer physically possible.
A research team led by Tsinghua University has proposed Transition Occupancy Matching as a common mathematical view of these two sources of change. A policy shift changes which action is selected in a state, while a dynamics shift changes the next state produced by that action. Both therefore change the joint distribution of state, action and next state, which the researchers call transition occupancy.
The team turned this principle into Occupancy-Matching Policy Optimization, or OMPO. Instead of treating all replayed data equally, OMPO maintains a large global buffer containing accumulated experience and a small first-in, first-out local buffer containing the most recent interactions. A discriminator compares transitions from the two buffers and estimates which historical transitions remain compatible with the current physical regime. Compatible experience is retained, while stale or mismatched transitions are downweighted.
OMPO also uses a sign-free logarithmic link to reformulate the matching objective as a stable min-max optimization. This design allows the method to work with reward signals that contain both bonuses and penalties, without adding a task-specific reward shift. A distributional critic models the full distribution of possible returns rather than only their mean, helping the agent account for randomness from action noise, sensing and contact dynamics. For visual tasks, the policy, critic and discriminator share a co-trained visual encoder, while an ODE-based flow actor represents potentially multimodal actions.
The researchers evaluated OMPO across three forms of distribution shift: policy shifts under stationary dynamics, transfer between tasks or domains, and policy shifts combined with non-stationary dynamics. Tests covered eight DeepMind Control locomotion tasks, a dog walk-to-run transfer, four MuJoCo environments with changing body dimensions, gravity and wind, Panda-Gym and Meta-World manipulation, and a real-robot proof of concept.
In DeepMind Control, OMPO learned faster and more stably than SAC and TD7 across both state-based and visual tasks. In the dog transfer experiment, the agent retained a replay buffer dominated by walking experience when the objective changed to running. Context-aware baselines struggled to adapt, while OMPO reweighted the old data and acquired the faster gait with substantially less negative transfer.
Under non-stationary MuJoCo dynamics, torso and foot lengths changed over episodes, while gravity and wind included stochastic changes during training. OMPO consistently outperformed CaDM and CEMRL. The results support the method's central premise: an agent does not need to explicitly identify every changed physical parameter if it can detect which transitions no longer match the current occupancy.
The manipulation results showed similar robustness. With action noise in Panda-Gym, OMPO achieved 98.4% success on Panda-Reach-Dense and 94.3% on Panda-Reach-Sparse. Across five reported contact-rich Meta-World tasks, it recorded the highest success rate in each. The largest margins appeared in coffee_push, where success reached 76.8% compared with the strongest baseline's 26.4%, and hammer, where it reached 78.3% compared with 32.8%.
The team also tested OMPO on a TianJi robot performing pick-and-place. In the disturbed condition, an object was moved before grasping, changing the outcome of the robot's nominal approach. Using recent interactions as the local anchor, the robot re-approached the displaced object, grasped it and completed the placement. The authors describe this as a proof-of-concept under a support-preserving disturbance, rather than a comprehensive hardware benchmark.
By providing one correction mechanism for both policy and dynamics shifts, OMPO offers a way to reuse costly historical experience without assuming that the physical world is stationary. The framework could complement vision-language-action models and world-model-based agents by providing an online post-training mechanism for continual adaptation.
The researchers note that OMPO still requires sufficient overlap between historical and current experience, as well as a local buffer that refreshes quickly enough to track the active physical regime. Support-breaking failures, extremely rapid shifts, actuator loss and severe hardware damage remain outside the current validation. Larger real-robot studies and stronger convergence and stability guarantees are important next steps.
The research was conducted by Yu Luo, Lei Lv, Fuchun Sun and Huaping Liu at Tsinghua University, Huawei Technologies' 2012 Laboratories and the Shanghai Research Institute for Intelligent Autonomous Systems at Tongji University. It was supported by the National Key Research and Development Program of China (2024YFB4711102). The OMPO implementation is publicly available on GitHub. The accepted manuscript was published online in National Science Review on 19 August 2026.
National Science Review
Computational simulation/modeling