wrist
Turning Video Models into
Generalist Robot Policies
1MIT2Independent Researcher3CMU4Amazon FAR*equal contribution
Double-click any video on this page to view it fullscreen.
A 14B video generative model, paired with a faithful inverse dynamics model, solves a wide range of robotics challenges — from zero-shot pick-and-place on a real-world Panda arm to contact-rich re-orientation of a cube with a 16-DoF multi-fingered hand.
We leave the video planner untouched and instead train an embodiment-specific inverse dynamics model (IDM) built on the robot’s Jacobian. The resulting closed-loop policy — VERA — translates dreamed futures into low-level actions faithfully, achieving zero-shot manipulation on a Panda arm and 16-DoF dexterous in-hand reorientation with the same planner. Decoupled video planning plus faithful translation is a viable path to generalizable robot control.
Figure 1. Controlling robots across embodiments, skills, and environments with a single video planner.
A planner, a translator, a closed loop
From a short observation history, the video model imagines a visual plan; the Jacobian IDM inverts each step of that plan into a chunk of low-level actions, the robot executes them, and observations return to the video model for the next round.
Figure 2. Translating video to actions. a, Given context frames, the video world model rolls out a short visual plan. b, The Jacobian IDM inverts each step of this path into a chunk of low-level actions. c, The chunk is executed and observations return to the video model for closed-loop execution.
Zero-shot manipulation on a real Panda
Six manipulation tasks on real hardware, specified only by language.
Prompt“place the yellow cube on top of the lego block.”
external 1
external 2
Left three: real-world execution · right three: dream rollouts.
Jacobian predictions.
Prompt“approach and push the blue button.”
wrist
external 1
Jacobian predictions.
Prompt“approach and push the orange button.”
wrist
external 2
Jacobian predictions.
Prompt“pick up the green tennis ball and place it inside the bowl.”
wrist
external 1
Jacobian predictions.
Prompt“place the yellow cube on top of the plate.”
Jacobian predictions.
Prompt“take the purple cube out of the plate and place it on the side.”
Jacobian predictions.
Reasoning under occlusion
The target hides behind a wall, visible from only one of three cameras, with a distractor (orange button) probing whether the planner truly grounds the language prompt.
Figure 3. The blue target is visible from only one of three viewpoints; an orange distractor probes whether the planner truly grounds the language prompt.
Prompt“push the blue button.”
VERA (ours)
✓ Succeedswrist
external 1
external 2
Jacobian predictions.
The video planner imagines a path around the wall; the Jacobian IDM translates that plan into an action chunk that finds and presses the hidden target.
DreamZero (baseline)
✗ Failsexternal 1
external 2
The dreamed future itself stops short of locating the hidden button — suggesting the bottleneck lies in the video branch, not in the action translation. This is consistent with the broader hypothesis: decoupling the planner from the action head can preserve visual reasoning that joint training may dilute.
Same scene, three goals
From the same initial observation, three language prompts steer the video planner to three different objects on the table. Prompt grounding is inherited intact from the video model.
Prompt“approach the cup.”
Prompt“approach the lego block.”
Prompt“approach the tennis ball.”
A 16-DoF hand, the same planner
We evaluate on dexterous multi-fingered reorientation with 16 DoFs, specifying the goal only by language. The video model is trained on cube-reorientation demonstrations paired with three prompts — clockwise, counter-clockwise, and random. At inference time, the prompt steers the hand: opposite prompts yield opposite cube movements from the same initial state.
Prompt“rotate the cube clockwise around the vertical axis.”
rollout (sim)
Jacobian
Prompt“rotate the cube counter-clockwise around the vertical axis.”
rollout (sim)
Jacobian
The same video planner that drives a 7-DoF arm also commands a real 16-DoF Allegro hand. Pairing one planner with an embodiment-specific Jacobian IDM is enough to translate the dreamed future into coordinated finger motion on hardware.
Prompt“rotate the cube clockwise around the vertical axis.”
real-world
dream
Jacobian
PushT: contact-rich planar pushing
A 2-DoF planar-pushing benchmark with contact-rich corner regrasps — a controlled setting where the gap between a faithful Jacobian translator and an unstructured baseline becomes directly measurable in closed-loop success.
run 1
run 2
run 3
A faithful IDM scales with action dimensionality
A controlled study isolating the design choice. Under matched data and DoF budgets, the Jacobian-parameterized IDM holds a better data–accuracy frontier than unstructured baselines — and the gap widens as embodiment dimensionality grows.
Figure 4. The Jacobian-parameterized IDM holds a better data–accuracy frontier as embodiment dimensionality grows.
Try it yourself
Code, checkpoints, and walkthrough notebooks are released: code on GitHub and models on Hugging Face. If you want to see the planner at work before setting up any robot or simulator, the DROID generation notebook continues the same real context frames under different language prompts, and the generated futures differ accordingly — the executed outputs ship in the notebook.
BibTeX
@article{li2026turningvideomodelsgeneralist,
title={Turning Video Models into Generalist Robot Policies},
author={Sizhe Lester Li and Evan Kim and Xingjian Bai and Tong Zhao and Tao Pang and Max Simchowitz and Vincent Sitzmann},
year={2026},
eprint={2605.27817},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2605.27817},
}
© 2026 The VERA Authors