Preprint · 2026

Turning Video Models into Generalist Robot Policies

Sizhe Lester Li^1,∗ Evan Kim^1,∗ Xingjian Bai^1,∗

Tong Zhao² Tao Pang² Max Simchowitz^3,4 Vincent Sitzmann¹

¹MIT ²Independent Researcher ³CMU ⁴Amazon FAR ^∗Equal contribution

†Double-click any video, or tap the button in its corner, to view fullscreen.

01Overview

A single video planner generalizes across embodiments.

A 14B video generative model, paired with a faithful inverse dynamics model, solves a wide range of robotics challenges — from zero-shot pick-and-place on a real-world Panda arm to contact-rich re-orientation of a cube with a 16-DoF multi-fingered hand.

Fig. 1Controlling robots across embodiments, skills, and environments with a single video planner.

02Abstract

Decouple planning. Translate faithfully.

We leave the video planner untouched and instead train an embodiment-specific inverse dynamics model (IDM) built on the robot's Jacobian. The resulting closed-loop policy — VERA — translates dreamed futures into low-level actions faithfully, achieving zero-shot manipulation on a Panda arm and 16-DoF dexterous in-hand reorientation with the same planner. Decoupled video planning plus faithful translation is a viable path to generalizable robot control.

03Method

A planner, a translator, a closed loop.

From a short observation history, a video generative model imagines a visual plan; the Jacobian IDM inverts each step of that plan into a chunk of low-level actions, the robot executes them, and observations return to the video model for the next round.

VERA method overview: video world model → Jacobian IDM → robot, in a closed loop — Fig. 2Translating video to actions. a, Given context frames, our video world model rolls out a short visual plan. b, The Jacobian IDM inverts each step of this path into a chunk of low-level actions. c, The chunk is executed and observations return to the video model for closed-loop execution.

04Real-world Panda · Zero-shot

Six manipulation tasks, specified only by language.

Prompt“place the yellow cube on top of the lego block.”

awrist

bexternal 1

cexternal 2

Left three: real-world execution · Right three: dream rollouts

Jacobian predictions

Prompt“approach and push the blue button.”

awrist

bexternal 1

Jacobian predictions

Prompt“approach and push the orange button.”

awrist

bexternal 2

Jacobian predictions

Prompt“pick up the green tennis ball and place it inside the bowl.”

awrist

bexternal 1

Jacobian predictions

Prompt“place the yellow cube on top of the plate.”

Jacobian predictions

Prompt“take the purple cube out of the plate and place it on the side.”

Jacobian predictions

05Reasoning · Occlusion

The target hides behind a wall — visible from only one of three cameras.

We start with occlusion-based reasoning: the target hides behind a wall, visible from only one of three cameras, with a distractor (orange button) probing whether the planner truly grounds the language prompt.

Prompt“push the blue button.”

VERA (ours)

✓ Succeeds

awrist

bexternal 1

cexternal 2

Jacobian predictions

The video planner imagines a path around the wall; the Jacobian IDM translates that plan into an action chunk that finds and presses the hidden target.

DreamZero (baseline)

✗ Fails

aexternal 1

bexternal 2

The dreamed future itself stops short of locating the hidden button — suggesting the bottleneck lies in the video branch, not in the action translation. This is consistent with the broader hypothesis: decoupling the planner from the action head can preserve visual reasoning that joint training may dilute.

06Prompt control

Same scene. Three different goals.

From the same initial observation, three language prompts steer the video planner to three different objects on the table. Prompt grounding is inherited intact from the video model.

Prompt“approach the cup.”

Prompt“approach the lego block.”

Prompt“approach the tennis ball.”

07Allegro Hand · Simulation · 16-DoF

Dexterous in-hand cube reorientation.

We evaluate our method on dexterous multi-fingered reorientation tasks with 16 DoFs, specifying the goal only by language prompt. The video model is trained on cube-reorientation demonstrations paired with three language prompts — clockwise, counter-clockwise, and random. At inference time, the language prompt steers the hand to produce the intended rotation: opposite prompts yield opposite cube movements from the same initial state.

Prompt“rotate the cube clockwise around the vertical axis.”

aRollout

bJacobian

Prompt“rotate the cube counter-clockwise around the vertical axis.”

aRollout

bJacobian

08Allegro Hand · Real world

Real-world dexterous reorientation.

The same video planner that drives a 7-DoF arm also commands a real 16-DoF Allegro hand. Pairing one planner with an embodiment-specific Jacobian IDM is enough to translate the dreamed future into coordinated finger motion on hardware.

Prompt“rotate the cube clockwise around the vertical axis.”

aReal-world

bDream

cJacobian

09PushT · Simulation

Push a T-block to a goal pose.

A 2-DoF planar-pushing benchmark with contact-rich corner regrasps — a controlled setting where the gap between a faithful Jacobian translator and an unstructured baseline becomes directly measurable in closed-loop success.

Run 1

Run 2

Run 3

10Designing a faithful IDM

J-IDM scales with action dimensionality.

A controlled study isolating the design choice. Under matched data and DoF budgets, the Jacobian-parameterized IDM holds a better data–accuracy frontier than unstructured baselines — and the gap widens as embodiment dimensionality grows.

J-IDM scales more favorably with action dimensionality — Fig. 4The Jacobian-parameterized IDM holds a better data–accuracy frontier as embodiment dimensionality grows.