Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Agent-to-Sim builds persistent 4D reconstructions of pets and humans from casual longitudinal smartphone RGBD videos via coarse-to-fine multi-video registration, then trains a hierarchical diffusion model of goals, paths, and body motion that is interactive with the scene and observer.
Problem
Learning interactive 3D behavior models of agents (animals, humans) from casually captured, long-span video collections without marker-based tracking or multiview studios, which requires persistent 3D correspondence across many videos.
Approach
A coarse-to-fine multi-video registration that uses a DINOv2-feature-based neural localizer for global camera alignment, followed by featuremetric bundle adjustment and scene-code swapping to build a complete persistent 4D representation; a hierarchical goal-path-pose diffusion model conditioned on egocentric perception (scene, observer, past motion).
Key results
- Full model achieves goal minADE 0.448 m vs 0.942 m for the Gaussian baseline and 0.663 m for location prior on the cat test split.
- Neural localizer plus featuremetric BA reduces camera rotation error to 6.35 degrees vs 37.59 without neural localizer and 59.19 for multi-video TotalRecon.
- 4D reconstruction DepthAcc (all) of 0.708 vs 0.533 for single-video TotalRecon and 0.099 for multi-video TotalRecon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the neural localizer (ablation) | ATS camera registration · Camera registration | Rotation Error (°): 37.59 -> 6.35 | Helped |
| add featuremetric bundle adjustment (ablation) | ATS camera registration · Camera registration | Rotation Error (°): 22.47 -> 6.35 | Helped |
| use ATS instead of multi-video TotalRecon | 4D reconstruction · Novel-view reconstruction | DepthAcc (all): 0.099 -> 0.708 | Helped |
| use a hierarchical goal-path-body model instead of a 1-stage motion model (ablation) | ATS behavior prediction · Cat behavior prediction | Goal (m): 1.322$^{\pm 0.071}$ -> 0.448$^{\pm 0.146}$ | Helped |
| use egocentric rather than world coordinates (ablation) | ATS behavior prediction · Cat behavior prediction | Goal (m): 1.164$^{\pm 0.043}$ -> 0.448$^{\pm 0.146}$ | Helped |
| add observer conditioning (ablation) | ATS behavior prediction · Cat behavior prediction | Goal (m): 0.647$^{\pm 0.148}$ -> 0.448$^{\pm 0.146}$ | Helped |
| add scene conditioning (ablation) | ATS behavior prediction · Cat behavior prediction | Goal (m): 0.784$^{\pm 0.126}$ -> 0.448$^{\pm 0.146}$ | Helped |
| use egocentric rather than world coordinates (ablation) | ATS spatial control · Spatial control | Path (m): 0.209$^{\pm0.002}$ -> 0.115$^{\pm0.006}$ | Helped |
| use ControlUNet rather than concatenating spatial-control and perception codes (ablation) | ATS spatial control · Spatial control | Path (m): 0.146 $^{\pm 0.005}$ -> 0.115$^{\pm0.006}$ | Helped |
| use a multimodal diffusion model instead of Gaussian distributions | ATS behavior prediction · Cat behavior prediction | Goal (m): 0.942$^{\pm 0.081}$ -> 0.448$^{\pm 0.146}$ | Helped |
| add scene annealing (ablation) | ATS multi-video reconstruction · Partial video captures | scene reconstruction quality | Helped |
| use ATS instead of Gaussian behavior control | ATS spatial control · Spatial control | Path (m): 0.206$^{\pm0.002}$ -> 0.115$^{\pm0.006}$ | Helped |
Surprising findings
- Multi-video TotalRecon performs worse than single-video TotalRecon on DepthAcc (all), indicating naive multi-video fitting fails without good registration.
Limitations
- Transient scene structures (moved cushions, new furniture) are not reconstructed well from few views.
- Only handles agent-observer interactions; multi-agent behavior is out of scope and the dataset is small (one month, 23 cat videos).
- Generated motion may be physically implausible (ground penetration, foot sliding) and the model learns only mid-level behaviors within a 6.4s horizon.
Experimental setup
Datasets: iPhone RGBD video collections of cat, human, dog, bunny
Benchmarks: cat held-out video behavior prediction camera registration with annotated correspondences novel-view 4D reconstruction on synchronized two-camera captures
Metrics: minADE minDE rotation error translation error DepthAcc LPIPS
Baselines: Location prior Gaussian MDM-style 1-stage model ActionMap-style world-coordinate model TotalRecon (single video) Multi-video TotalRecon
Builds on: TotalRecon DINOv2 GMD MDM dynamic 3D Gaussian splatting
Closest papers by meaning
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.86)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
- iVideoGPT: Interactive VideoGPTs are Scalable World Models (NeurIPS 2024 · similarity 0.86)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.86)
- SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning (NeurIPS 2026 · similarity 0.85)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.