TL;DR: We factor action realization and robot appearance out of the world model, presenting actions as visible robot geometry and leaving the model to learn how the scene responds.

Abstract

Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene subsequently responds through contact outcomes and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose robot-factored world models, which move two robot-specific factors outside the world model. First, action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory. This middle signal avoids both robot-specific action-realization learning and future-state leakage. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. To resolve depth ambiguity in the rendered interface, we further pair end-effector depth with scene depth, providing geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered robot geometry form a shared visual world-model interface that remains consistent across viewpoints and robot embodiments. On DROID and RoboCasa-GR1, the rendered interface outperforms vector-conditioned baselines. We further demonstrate that the same rendered interface supports unseen robot embodiments at inference, and, as a downstream application, generate robot-interaction videos from retargeted human demonstrations.

Results

DROID

Fixed exterior camera · robot arm with parallel gripper

AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT

RoboCasa-GR1

Egocentric moving camera · humanoid with dexterous hands

AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT
AdaLNvector-based conditioningOursrendering-based conditioningGT

The rendered interface gives the video model direct pixel-space evidence of robot motion, so predicted scene changes are better localized around the robot and the contact region than vector- or pose-conditioned baselines. Rendered robot geometry localizes robot-driven scene changes, while depth helps resolve contact-relevant proximity and occlusion.

Action Controllability

Holding the initial scene fixed and editing only the rendered nominal motion changes the predicted scene response — the model uses rendered robot geometry as the action signal.

Original actionEdited action
Rendered actionPrediction

The edited trajectory is redirected toward a different object in the same scene.

Original actionEdited action
Rendered actionPrediction

The edited trajectory is re-solved to approach from higher above the object.

Zero-Shot Embodiment Generalization

Since an action enters only as rendered robot geometry, robots never seen during training — a new arm–hand pairing, or even multiple arms — pass through the exact same interface and still drive a plausible scene response. This is nontrivial for vector- or pose-conditioned models, where a new embodiment brings a different action space and a new action-to-motion mapping to learn. Each clip shows Mesh rendering (input) · Ours (generated) · GT.

Beyond the rendered mesh shown here, the model also conditions on scene depth and end-effector depth; both are omitted from these strips for space.

Unseen Composition — Robot Arm + Dexterous Hand

An xArm 6 arm paired with an Inspire F1 hand — a combination never seen together in training. Evaluated on HRDexDB.

Mesh renderingOursGT
Multiple Robot Arms — Dual Robot Arms

Two robot arms composed into a single scene — a robot count outside the single-arm training setup. Evaluated on DexMimicGen.

Mesh renderingOursGT

Application: Human Demonstration → Robot Video

DexYCB human demonstrationOurs — generated robot rollout
DexYCB human demonstrationOurs — generated robot rollout

Method Overview

Robot-factored visual world-model interface
Visual world-model interface. Static context carries scene and viewpoint; rendered nominal robot geometry carries action; the diffusion model predicts the scene response.

Instead of conditioning the world model on raw action commands, we factor out two robot-specific steps as fixed preprocessing. First, each action is rolled through the robot’s own controller and kinematics into a nominal trajectory — robot-only motion before any scene interaction. Second, this trajectory is rendered through the robot URDF into camera-aligned robot mesh RGB and end-effector depth. Paired with a camera-aware static stream (scene appearance and depth), these become the model’s entire action interface, leaving it the single shared problem of predicting how the scene responds.

Nominal Trajectory Conditioning

The right action signal lives between the raw command and the logged state: the controller-realized nominal trajectory is available at deployment and does not leak scene interaction.

(a) Three action signals — which one should condition the world model?
Raw action
Nominal trajectory
Realized state

DROID — the same episode rendered from each of the three signals.

Nominal trajectory — the motion the robot is expected to follow under the commanded action in the absence of any scene interaction, obtained by rolling the command through the robot's own controller and kinematics.

A raw action is a command, not a motion. It has not yet passed through the controller, so it reflects neither the controller's tracking behavior nor the robot's kinematic and actuation limits — and it therefore departs substantially from the state the robot actually reaches.

The realized state is recorded in every dataset, but it is unavailable at inference: it is precisely the future the world model is asked to predict. Conditioning on it leaks the interaction outcome, since contact, compliance, and latency are already baked into the logged state.

(b) The nominal–realized gap — what should the world model learn?
Nominal trajectory
Realized state

RoboCasa-GR1 — nominal trajectory and realized state on a contact-rich rollout.

The two coincide while the robot moves freely, and separate once it meets the scene, where contact and compliance pull the realized motion away from the nominal plan. That divergence is not noise in the conditioning signal — it is the interaction itself, and predicting it is exactly what we leave to the world model.

Impact of Depth Conditioning

Without depthWith depthGT
Without depthWith depthGT

RGB mesh rendering places the robot only in the image plane, where overlap alone cannot tell a real touch from a robot simply passing in front of or behind an object. We pair end-effector depth with scene depth to make the model depth-aware, avoiding false contact from image-plane overlap.

BibTeX

@article{kim2026rofacto,
  title   = {Robot-Factored World Models via Robot Rendering},
  author  = {Kim, Byungjun and Kim, Taeksoo and Cha, Hyunsoo and Joo, Hanbyul},
  journal = {arXiv preprint arXiv:2607.22535},
  year    = {2026}
}