Research map / 3D vision / 3D generation
Learning Fused State Representations for Control from Multi-View Observations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MFSC, a multi-view RL method that fuses per-view CNN embeddings with a learnable [state] token via self-attention, trained with a bisimulation-based cosine metric loss plus a masked latent reconstruction auxiliary task, and evaluated on Meta-World, PyBullet Ant, DMControl and CARLA.
Problem
Learning compact, task-relevant fused state representations from multi-view observations for control, robust to redundancy, distracting views, and missing views.
Approach
Integrates bisimulation metric learning (cosine-distance base metric with a latent dynamics ensemble) into a self-attention multi-view fusion module with a learnable fusion token, plus a masked latent reconstruction auxiliary loss with a learnable mask token for missing views.
Key results
- On Meta-World, MFSC outperforms MVRL baselines (Keypoint3D, LookCloser, F2C, MVD) across five manipulation tasks with higher sample efficiency and stability.
- On CARLA, MFSC achieves episode return 168.01±9.62 vs SAC 152.73±4.54, but shows no significant improvement over SAC overall.
- On DMControl (single-view degraded, 1M steps), MFSC trails TACO on most tasks (e.g., Cheetah Run 679 vs 915) and is competitive with Dreamer-v3 and TD-MPC2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFSC instead of the strongest listed baseline | DMControl single-view setting · Cartpole Swingup (1M Steps) | return: 865±15 -> 832±29 | Hurt |
| Use MFSC instead of the strongest listed baseline | DMControl single-view setting · Cheetah Run (1M Steps) | return: 915±6 -> 679±33 | Hurt |
| Use MFSC instead of the strongest listed baseline | DMControl single-view setting · Finger Spin (1M Steps) | return: 985±2 -> 977±7 | Hurt |
| Use MFSC instead of the strongest listed baseline | DMControl single-view setting · Walker walk (1M Steps) | return: 968±3 -> 909±40 | Hurt |
| Use MFSC instead of SAC | CARLA · CARLA | Episode return: 152.73±4.54 -> 168.01±9.62 | Helped |
| Use MFSC instead of SAC | CARLA · CARLA | Successes episodes: 21% -> 13% | Hurt |
| Add the bisimulation constraint (ablation) | MFSC · Meta-World and PyBullet Ant | returns | Helped |
| Add multiview masking and latent reconstruction (ablation) | MFSC · Meta-World and PyBullet Ant | returns | Helped |
| Replace feature averaging with self-attention fusion (ablation) | MFSC · Meta-World and PyBullet Ant | returns | Helped |
| Use all three views instead of a single view (ablation) | MFSC · Meta-World | performance | Helped |
| Add a task-irrelevant view | MFSC · Meta-World noisy-view setting | performance | No clear effect |
Surprising findings
- On CARLA, the proposed method does not significantly beat the SAC concatenation baseline, which the authors attribute to near-independent information across views.
- MFSC has a lower successes-episodes rate than SAC on CARLA (13% vs 21%).
Limitations
- Reliance on masking to handle missing views; reconstructing crucial views is challenging when view information is incomplete.
- Performance gap to SOTA TACO on DMControl and no significant gain over SAC on CARLA.
- Bisimulation metric is approximated with a cosine base metric, so the theoretical bound relies on an approximation.
Experimental setup
Datasets: Meta-World PyBullet Ant DMControl CARLA DMControl-GB video_hard distractors
Benchmarks: Meta-World PyBullet Ant Ant No Color DeepMind Control Suite CARLA
Metrics: episode return sample efficiency success rate distance collision intensity
Baselines: Keypoint3D LookCloser Fuse2Control (F2C) MVD RAD Vanilla PPO SAC CURL DrQ-v2 TACO Dreamer-v3 TD-MPC2
Builds on: bisimulation metrics (DBC) MICo SimSR SimSiam Keypoint3D
Closest papers by meaning
- Learning Latent Dynamic Robust Representations for World Models (ICML 2024 · similarity 0.89)
- SeqMvRL: A Sequential Fusion Framework for Multi-view Representation Learning (CVPR 2025 · similarity 0.87)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.87)
- DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning (CVPR 2024 · similarity 0.86)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.