Research map / Vision-language and multimodal / Cross-modal learning
DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMR, a three-branch representation learning framework that decomposes RGB and DVS event inputs into shared co-features and modality-specific noise for visual RL on a new CARLA benchmark.
Problem
Fusing frame-based RGB and event-based DVS observations for visual reinforcement learning, where only reward signals (no pixel-level supervision) are available, tends to inject task-irrelevant noise into the latent state.
Approach
Explicitly decompose the latent representation into co-features shared across modalities and per-modality noise components; train the noises with reconstruction losses, push them away from co-features with a contrastive loss, and drive co-features with DeepMDP reward and state prediction heads, using only the co-feature branch at test time.
Key results
- On the Carla benchmark, DMR reaches reward 194±73 in JW-Midnight and 112±51 in HB-Hardrain, outperforming the multi-modality baselines TransFuser, EFNet, FPNet, and RENet.
- DMR outperforms all single-modality DeepMDP policies (RGB, DVS, DVS-F) by a substantial margin in most scenarios, except HB-Midnight where gains are limited.
- Ablation: moving from a single-branch fusion to the full three-branch model with contrastive repel and reconstruction raises JW-Midnight reward from 145 to 194.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two modality branches and concatenate their features for policy learning instead of a single co-feature branch (ablation) | DMR · JW-Midnight | Distance: 185±55 -> 194±85 | No clear effect |
| Use two modality branches and concatenate their features for policy learning instead of a single co-feature branch (ablation) | DMR · JW-Midnight | Reward: 145±54 -> 141±78 | No clear effect |
| Add three-branch decomposition and contrastive repulsion (ablation) | DMR · JW-Midnight | Distance: 194±85 -> 214±74 | Helped |
| Add three-branch decomposition and contrastive repulsion (ablation) | DMR · JW-Midnight | Reward: 141±78 -> 181±77 | Helped |
| Add reconstruction decoder to the three-branch model with contrastive repulsion (ablation) | DMR · JW-Midnight | Distance: 214±74 -> 230±77 | Helped |
| Add reconstruction decoder to the three-branch model with contrastive repulsion (ablation) | DMR · JW-Midnight | Reward: 181±77 -> 194±73 | Helped |
| Use DMR with an SNN DVS branch (ablation) | DMR · HB-Hardrain | Distance: 101±10 -> 143±30 | Helped |
| Use DMR with an SNN DVS branch (ablation) | DMR · HB-Hardrain | Reward: 45±2 -> 117±34 | Helped |
| Fuse RGB and LiDAR using DMR instead of LiDAR alone (ablation) | DMR · HB-Hardrain | Distance: 113±9 -> 145±19 | Helped |
| Fuse RGB and Depth using DMR instead of Depth alone (ablation) | DMR · HB-Hardrain | Distance: 103±12 -> 109±25 | Helped |
| Use DMR rather than RENet | DMR · JW-Midnight | Reward: 158±101 -> 194±73 | Helped |
| Use DMR rather than RENet | DMR · JW-Hardrain | Reward: 6±30 -> 111±57 | Helped |
Surprising findings
- Existing SOTA multi-modality fusion methods (TransFuser, EFNet, FPNet, RENet) often fail to match the best single-modality policies in the CARLA benchmark.
- In HB-Hardrain, RGB alone outperforms DVS, as raindrops produce excessive DVS noise.
Limitations
- Evaluated only in simulated CARLA driving scenarios; generalization to real-world sim2real transfer is future work.
- Limited improvement in the HB-Midnight scenario.
- Reported results come from 3 seeds, and some differences are within large standard deviations.
Experimental setup
Datasets: CARLA simulator RGB and DVS observations
Benchmarks: Carla benchmark (JW-Midnight, JW-Hardrain, HB-Midnight, HB-Hardrain)
Metrics: episode reward distance
Baselines: DeepMDP single-modality RGB DeepMDP DVS DeepMDP DVS-F TransFuser EFNet FPNet RENet
Builds on: DeepMDP Soft Actor-Critic TransFuser EFNet FPNet RENet
Closest papers by meaning
- Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL (NeurIPS 2024 · similarity 0.91)
- A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations (NeurIPS 2024 · similarity 0.88)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.87)
- VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning (CVPR 2025 · similarity 0.87)
- Spotlight on Token Perception for Multimodal Reinforcement Learning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.