Research map / Vision-language and multimodal / Cross-modal learning
Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dissected Dynamics Modeling (DDM), which splits multi-modal visual RL features into modality-consistent and inconsistent parts and models their dynamics via separate pathways, evaluated on CARLA and DMControl.
Problem
Environment dynamics modeling for visual RL with multiple input modalities is hard because cross-modal relationships are heterogeneous and volatile; existing single-modality dynamics models (e.g., DeepMDP) do not handle this.
Approach
Decompose each modality's features into modality-consistent (via mutual and transition prediction across modalities) and modality-inconsistent parts (via cross-modality orthogonality), route them into separate dynamics pathways, and add a reward predictive function to filter task-irrelevant information.
Key results
- On CARLA (RGB+event), DDM achieves average episode return 272.75 vs 216.25 for DeepMDP and 221.75 for SPR, roughly 24% over DeepMDP.
- On DMControl, DDM averages 781.67 on the easy group vs 724.67 for DeepMDP, and 508.67 vs 458.33 for SPR on the hard group.
- Ablation on CARLA shows each component (Lfp, Lr, Lmp, Ltp, Lorth) adds gains, with the orthogonality objective giving the largest final improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DDM instead of HAVE | SAC-based multimodal visual RL · CARLA, 2 modalities, average | episode return: 245.50 -> 272.75 | Helped |
| use DDM instead of HAVE | SAC-based multimodal visual RL · CARLA, 2 modalities, average | driving distance (m): 275.25 -> 304.75 | Helped |
| use DDM instead of SPR | SAC-based multimodal visual RL · DMControl, Cartpole Swingup, Reacher Easy, Cheetah Run average | episode return: 777.00 -> 781.67 | Helped |
| use DDM instead of SPR | SAC-based multimodal visual RL · DMControl, Cartpole Swingup_sparse, Reacher Hard, Quadruped Run average | episode return: 458.33 -> 508.67 | Helped |
| add full-state prediction objective Lfp (ablation) | SAC baseline on CARLA · CARLA, average | episode return: 164.25 -> 207.50 | Helped |
| add reward prediction objective Lr (ablation) | SAC + Lfp on CARLA · CARLA, average | episode return: 207.50 -> 219.75 | Helped |
| add mutual prediction objective Lmp (ablation) | SAC + Lfp + Lr on CARLA · CARLA, average | episode return: 219.75 -> 232.75 | Helped |
| add transition prediction objective Ltp (ablation) | SAC + Lfp + Lr + Lmp on CARLA · CARLA, average | episode return: 232.75 -> 253.00 | Helped |
| add cross-modality inconsistency orthogonality objective Lorth (ablation) | SAC + Lfp + Lr + Lmp + Ltp on CARLA · CARLA, average | episode return: 253.00 -> 272.75 | Helped |
| replace cross-modality inconsistency orthogonality with same-modality consistency–inconsistency orthogonality (Eq. 6) (ablation) | DDM on CARLA · CARLA, Normal, Midnight, Dazzling, Rainy | driving distance: 305, 338, 291, 300 -> 249, 267, 299, 314 | Hurt |
| use DDM instead of HAVE | SAC-based multimodal visual RL · CARLA, 4 modalities, average | episode return: 283.75 -> 303.00 | Helped |
| use DDM instead of HAVE | SAC-based multimodal visual RL · CARLA, RGB + LiDAR-BEV, average | episode return: 246.25 -> 268.25 | Helped |
Surprising findings
- Enforcing orthogonality within the same modality (Eq. 6) performs worse than the cross-modality inconsistency constraint, suggesting over-strong regularization hurts.
Limitations
- No integrated planning or world-model mechanism for long-term action consequences.
- Focuses only on vision-based inputs; does not cover text or audio modalities.
- Multi-modal experiments are on simulated benchmarks with synthetic masking used to create inconsistency on DMControl.
Experimental setup
Datasets: CARLA simulator (RGB, event, depth, LiDAR BEV) DeepMind Control Suite (RGB, depth)
Benchmarks: CARLA DMControl
Metrics: episode return driving distance (m) reward estimation error state prediction error
Baselines: SAC DrQ DeepMDP SPR TransFuser EFNet MuMMI MAIE HAVE
Builds on: DeepMDP Soft Actor-Critic SPR DrQ
Closest papers by meaning
- DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning (CVPR 2024 · similarity 0.91)
- DyMoDreamer: World Modeling with Dynamic Modulation (NeurIPS 2025 · similarity 0.89)
- A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations (NeurIPS 2024 · similarity 0.88)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.