Latent Jacobian Steer

Eliciting Contact-Rich Out-of-Distribution Behaviors from Pretrained Humanoid Controllers

Anonymous authors

LJS discovers new physical interactions by mapping joint-action proposals into a frozen controller’s latent space and evaluating their effects in simulation.

New

From one search rollout to a deployable policy

Can what LJS discovers become a controller for a real robot? We gave it a new task, a Unitree G1 climbing onto a 0.75 m table pushed against a wall and standing on it, and trained a closed-loop policy from a single CEM-LJS rollout. Writing the physical effects, searching and training took 3 to 4 hours in total.

The policy uses only what the robot measures itself: its IMU, its joint encoders, and the edge of the table as seen by its head camera. It climbs onto the table and stands up in 96% of 512 simulated trials, with the table’s position, height and friction, the robot’s motor gains and masses, and every sensor reading randomized. It also carries over without retraining to a second simulator, standard MuJoCo, where it climbed and stood on all 48 randomized tables we tried. That is the engine running the live demo below.

1CEM-LJS rollout
96%of 512 randomized trials
45 minof training on one GPU
On-robotsensors only
One policy, four randomized tables.

Run the policy in your browser

Pick a table, deploy the policy and watch it climb. The physics and the policy both run live on your device, with nothing prerecorded. The policy sees only what the real robot would.

About 75 MB, loaded once
Speed

Load the simulation to begin.

The sliders cover the ranges the policy was trained on. Drag to orbit the camera.

Hardware demonstration coming soon.

Contact-rich behaviors without policy retraining

Using LJS, a frozen BFM-Zero humanoid controller traverses a hurdle, lifts and passes beneath a shutter, and opens a door with a carried cube. These behaviors are discovered through simulator search, without task-specific policy training or reference motions.

Playback speed

The key idea: steer through the policy Jacobian

Joint-action perturbations provide a direct way to express local control changes, while latent commands retain the structure of the pretrained policy. LJS connects these spaces through a damped local inverse of the policy Jacobian, allowing search in joint-action coordinates while executing through the frozen controller.

CEM proposes joint-action perturbations; a damped policy-Jacobian inverse maps them into latent commands; simulated physical effects guide search.
Method overview. Joint-action perturbations are mapped to latent commands through the policy Jacobian. Simulator rollouts evaluate progress toward the specified physical effect.

Propose joint-action changes

CEM samples changes to the G1 controller’s 29 joint targets. Simulated outcomes rank these proposals, and the strongest candidates guide the next round of sampling.

Map into latent commands

A damped inverse of the policy Jacobian maps each proposal into a 256-dimensional latent change. The updated command is normalized for execution through the frozen controller.

Simulate and replan

Using live observations, the frozen policy generates each rollout. Search executes the best valid command and replans from the new state until the physical effect is achieved.

Action-space proposals compensate for unequal sensitivity

A small latent change can move some combinations of joints strongly and others barely at all. Even searching the Jacobian’s own 29-dimensional latent subspace leaves this imbalance intact. LJS starts with comparable action changes and uses the local inverse to account for the controller’s sensitivity.

Write the policy Jacobian as J = UΣVT, with singular values si. For equal-variance Gaussian proposals with variance σ2, the local action variance along direction i is:

Sampling in the Jacobian’s latent subspace

σ2si2

Exploration is dominated by directions to which the controller is most sensitive.

LJS action-space pullback

σ2(si2si2+λ2)2

When si is large relative to λ, the variance approaches σ2. Damping suppresses weakly responsive directions.

LJS therefore gives realizable action directions comparable exploration while limiting excessive latent changes. These directions can combine multiple joints. The argument concerns the policy’s local response; simulator rollouts determine which proposals produce useful contact-rich motion.

Tasks are specified through ordered physical effects, such as establishing support, raising an object, or clearing an obstacle. Each effect has a score that guides search and an acceptance condition that determines completion. The result is a sequence of latent commands. During execution, the frozen policy continues to receive updated observations and generate joint targets, retaining its feedback response throughout the interaction.

Coupling action and latent spaces improves task completion

Across ten unseen seeds, with the same effects and settings for every method, CEM-LJS gets furthest on every task, and it is the only one that gets through the shutter.

MethodHurdleShutterDoor + cube
All search times
Candidate steps needed to reach each physical effect for each method on the hurdle, shutter, and door with cube tasks.
Simulated candidate steps to reach each effect, including runs that stopped early. Lower is better.
Search ablations

The gain comes from coupling the two spaces, not from the optimizer or from searching fewer dimensions. Steering through the Jacobian generally beats searching the latent directly, and a random latent subspace of the same size gets nowhere. Rotating the joint-action axes changes little, while searching the Jacobian’s own latent directions without the rescaling of LJS completes less than half as many tasks. No baseline gets through the shutter at any proposal scale we tried.

SearchShutterHurdleDoor + cube
Joint-action proposals through the Jacobian
CEM-LJS7 / 103 / 1010 / 10
CEM-LJS, rotated joint-action axes7 / 100 / 1010 / 10
MPPI with LJS4 / 102 / 109 / 10
LATTICE-style LJS3 / 101 / 107 / 10
Zero-mean refinement LJS2 / 101 / 102 / 10
Targeted-mean refinement LJS1 / 100 / 104 / 10
Gaussian LJS0 / 101 / 102 / 10
Proposals in the latent space
Direct latent CEM0 / 101 / 106 / 10
Direct latent MPPI3 / 100 / 106 / 10
Direct latent Gaussian0 / 100 / 104 / 10
Direct latent targeted refinement0 / 100 / 103 / 10
Jacobian’s latent directions, without the rescaling3 / 100 / 106 / 10
Random 29-dimensional latent subspace0 / 100 / 100 / 10

Sanity check of the search methods with minimal definitions

Our main tasks use detailed effect definitions. To make sure the comparison does not depend on how those are written, we also built a task whose definition leaves nothing to tune. The robot carries a 3 kg box, held between its palms, along a curve of points in 3D. Each step only asks the box to reach the next point, and the box must never touch the floor. Nothing says how to grip, lift or walk.

The three paths were generated rather than designed. Every method runs five seeds on each path and may retry a failed step from an earlier state, up to 15 minutes per run.

The three paths, seen from above and at an angle. Each starts where the box rests between the robot’s palms and curves away to the left or right at heights between about 0.9 and 1.2 m.
MethodPaths completedPoints reached on average
CEM-LJS12 / 159.1 / 10
Direct latent CEM2 / 152.5 / 10
Residual-action CEM3 / 154.8 / 10
Mixed-LJS1 / 151.5 / 10

Each column shows CEM-LJS and one baseline on the same path and seed.

CEM-LJS, path 1

Mixed-LJS, path 1

CEM-LJS, path 2

Direct latent CEM, path 2

CEM-LJS, path 3

Residual-action CEM, path 3

Beyond completion: how the robot moves

Precise contacts must work together with whole-body coordination. These comparisons show how each method establishes support, manipulates objects, and moves through the task, using the best run of each method.

Measured on the same effects

Residual actions can push the robot through an effect, but they fight the coordination the controller already has. Replayed with the G1’s real joint and motor limits, CEM-LJS moves more smoothly, uses less power and slips less, and BFM-Zero’s own motion discriminator rates it as more human.

Per effectCEM-LJSResidual-action CEM
Mechanical power W/kg1.62.6
Joint jerk rad/s³7821,660
Time at the torque limit6%20%
Foot slip under load m/s0.020.11
Human-likeness−0.9−2.4

Shaping motion at its root

Because a task is just a set of physical-effect scores, we can shape how the robot moves inside the search itself. Adding BFM-Zero’s motion discriminator to the hurdle’s scores gives a calmer, more human-like crossing, at the cost of a longer search.

Per hurdle effectWithoutWith discriminator
Mechanical power W/kg1.40.9
Joint jerk rad/s³841633
Human-likeness−1.8−0.8
Seeds completed10 / 108 / 10

Without the discriminator

With BFM-Zero’s discriminator

Motion sequences

Hurdle traversal

Seven frames from the best CEM-LJS, direct latent CEM, and residual-action CEM runs on the hurdle.

Shutter traversal

Seven frames from the best CEM-LJS, direct latent CEM, and residual-action CEM runs on the shutter.

Door opening with a cube

Seven frames from the best CEM-LJS, direct latent CEM, and residual-action CEM runs on the door with a cube.

Reliable from new starts and under varied physics

New starting positions

The effect definitions don’t need retuning when the setup changes. From new starting positions and headings, CEM-LJS still gets through the shutter most of the time, while the baselines rarely do.

MethodFull tasks completed
CEM-LJS8 / 10
Direct latent CEM1 / 10
Residual-action CEM0 / 10
Mixed-LJS2 / 10

Turned 11°

Turned 12°

Moved 0.19 m

Retries and randomized physics

This setting is for practitioners who want success on every seed. A CEM-LJS search can occasionally settle in a poor local basin, so we retry a failed effect from an earlier state and score every candidate under varied friction, mass, motor strength and control delay. CEM-LJS then completes every task on every seed, in at most 15 minutes.

MethodHurdleShutterDoor + cube
CEM-LJS10 / 1010 / 1010 / 10
Direct latent CEM6 / 106 / 1010 / 10
Residual-action CEM7 / 100 / 102 / 10
Mixed-LJS2 / 103 / 1010 / 10
CEM-LJS mean search time8.0 min5.5 min2.4 min

Hurdle traversal

Shutter traversal

Door opening with a cube

Dexterous assembly with a frozen hand policy

The same search also steers a frozen Play2Perfect hand policy to assemble, screw and insert parts, then release them in place, without fine-tuning it for any of these tasks, and does as well as or better than direct goal search on every one.

MethodBeam, first partBeam, second partScrewingInsertion
All search times
Candidate steps needed to reach each physical effect for CEM-LJS and direct goal CEM on the four assembly tasks.
Simulated candidate steps to reach each effect, including runs that stopped early. Lower is better.

How the hand assembles

These comparisons show how each method grasps, aligns, and releases the part.

Motion sequences

Across controllers and embodiments

All LJS needs is a frozen policy that takes a latent or a goal, and a simulator. We also use it with ASE, FB-MEBE and SPiRL, and run BFM-Zero in a second simulator.

Explore every task, method, and seed

All 200 runs, ten seeds per task and method.