Research map / Video / Video-language understanding
DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaMind, which abstracts video into compact dynamic representations via an adaptive FrameScorer, predicts future dynamics with a transformer, and decodes actions with a dynamic-guided policy for language-conditioned embodied control.
Problem
Embodied decision-making from language instructions is hard because concise language commands do not map well onto rich, redundant video content, and frame-by-frame video generation leads to rigid mappings.
Approach
Instead of refining language or generating frame-by-frame video, the method learns a FrameScorer that weights frames by semantic consistency and visual saliency, fuses them into window-level dynamic representations, autoregressively predicts future dynamics conditioned on language, and uses them to guide an action transformer trained end-to-end.
Key results
- On LOReL Sawyer, DynaMind averages 53.67% success across tasks versus 43% for SkillDiffuser and 40% for LISA.
- On Franka Kitchen, DynaMind reaches 39.81% success versus 32.94% for GR-1 and 28.69% for LISA.
- In real-world experiments over 10 trials across 5 tasks, DynaMind averages 0.94 success versus 0.82 for LISA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive FrameScorer weighting instead of equal frame weights (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) and Kitchen | success rate | Helped |
| Use adaptive FrameScorer weighting instead of random frame weights (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) and Kitchen | success rate | Helped |
| Use window-level dynamic representations instead of selecting one random frame per window (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) and Kitchen | success rate | Helped |
| Set reasoning interval to a moderate value rather than C=5 or C=100 (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) | success rate | Helped |
| Use dynamic-guided action decisions rather than language-guided decisions (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) | success rate | Helped |
| Combine dynamic and language guidance for action decisions rather than using dynamic guidance alone (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) | success rate | Hurt |
| Include historical actions in the action decision module (ablation) | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) | success rate | Helped |
| Combine DynaMind with LISA language decomposition | DynaMind · Franka Kitchen | success rate: 39.81% -> 33.19% | Hurt |
| Abstract videos into dynamic representations and reason over future dynamics | DynaMind · LOReL Sawyer: average over tasks | success rate: 43% -> 53.67% | Helped |
| Abstract videos into dynamic representations and reason over future dynamics | DynaMind · BabyAI (GoToSeq, SynthSeq, BossLevel) | success rate: LISA: 59.4%, 46.3%, 49.1%; DynaMind: 72.7%, 50.7%, 52.3% | Helped |
| Transfer learned dynamic representations from GoToSeq to unseen tasks | DynaMind · BabyAI: unseen SynthSeq and BossLevel | success rate: SynthSeq: 33.1% -> 40.0%; BossLevel: 32.4% -> 35.7% | Helped |
| Use abstracted dynamic representations for unseen compositional tasks | DynaMind · LOReL Sawyer: unseen compositional tasks | success rate: 25.21% -> 36.67% | Helped |
Surprising findings
- Combining DynaMind with the language-decomposed LISA did not improve performance (33.19% vs 39.81% for DynaMind alone), which the authors attribute to information loss in language decomposition.
Limitations
- Uses fixed dynamic reasoning intervals, which the authors note can be suboptimal.
- Gains on BabyAI are attributed mainly to other modules since the FrameScorer contributes less in simple environments.
Experimental setup
Datasets: LOReL Sawyer Franka Kitchen BabyAI
Benchmarks: LOReL Sawyer Franka Kitchen BabyAI real-world Franka Research 3 tasks
Metrics: success rate
Baselines: Vanilla BC DT GR-1 MT-R3M LISA SkillDiffuser RL
Builds on: SkillDiffuser LISA GR-1 Decision Transformer R3M
Closest papers by meaning
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.89)
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning (NeurIPS 2025 · similarity 0.88)
- CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning (CVPR 2026 · similarity 0.88)
- V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning (ICML 2026 · similarity 0.88)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.87)
- VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.