Research map / Vision-language and multimodal / Cross-modal learning
Layer-Wise Modality Decomposition for Interpretable Multimodal Sensor Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Layer-Wise Modality Decomposition (LMD), a post-hoc method that linearizes a pretrained camera-radar/LiDAR BEV fusion network so each layer's output splits exactly into camera, radar, and bias contributions.
Problem
Interpreting how each sensor modality contributes to a prediction in multi-sensor autonomous-driving perception models is hard because modality information becomes entangled inside the fusion network.
Approach
Linearize activation and normalization layers of a pretrained fusion model (ratio-based slope for activations, stored statistics for LayerNorm/BatchNorm, bias split by rules) so that the network output decomposes exactly into per-modality terms across all layers, without changing the architecture; separation is checked with perturbation-based Pearson/MSE metrics.
Key results
- With identity BatchNorm and ratio-rule LayerNorm linearization, the radar-perturbed radar prediction reaches Pearson correlation 0.05 with its original output while the camera prediction stays at 1.00 (radar-camera setting).
- In the LiDAR-camera setting, the same LMD configuration gives radar-perturbed radar correlation 0.09 and unperturbed camera correlation 1.00.
- LMD+SHAP is reported to better satisfy modality-level separability than SHAP alone in modality-replacement experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LMD with the ratio rule instead of the baseline decomposition | SimpleBEV radar-camera fusion · Radar + Camera modality replacement | Pearson correlation Rp/R: 0.22 $\pm$ 0.10 -> \textbf{0.05} $\pm$0.02 | Helped |
| Apply LMD with the ratio rule instead of the baseline decomposition | SimpleBEV radar-camera fusion · Radar + Camera modality replacement | Pearson correlation Rp/C: 0.76 $\pm$ 0.07 -> \textbf{1.00} $\pm$ 0.00 | Helped |
| Apply LMD with the ratio rule instead of the baseline decomposition | SimpleBEV LiDAR-camera fusion · LiDAR + Camera modality replacement | Pearson correlation Rp/R: 0.41 $\pm$ 0.08 -> \textbf{0.09} $\pm$ 0.03 | Helped |
| Apply LMD with the ratio rule instead of the baseline decomposition | SimpleBEV LiDAR-camera fusion · LiDAR + Camera modality replacement | Pearson correlation Rp/C: 0.77 $\pm$ 0.05 -> \textbf{1.00} $\pm$ 0.00 | Helped |
| Use the ratio rule for LayerNorm rather than the uniform rule (ablation) | LMD on SimpleBEV radar-camera fusion · Radar + Camera modality replacement | Pearson correlation Cp/C: 0.38 $\pm$ 0.04 -> \textbf{0.15} $\pm$ 0.04 | Helped |
| Use the ratio rule for LayerNorm rather than the identity rule (ablation) | LMD on SimpleBEV LiDAR-camera fusion · LiDAR + Camera modality replacement | Pearson correlation Cp/C: 0.60 $\pm$ 0.07 -> \textbf{0.44} $\pm$ 0.06 | Helped |
| Use uniform rather than identity bias splitting for BatchNorm (ablation) | LMD on SimpleBEV radar-camera fusion · Radar + Camera modality replacement | Pearson correlation Rp/R: 0.15 $\pm$ 0.22 -> 0.50 $\pm$ 0.04 | Hurt |
| Use uniform rather than identity bias splitting for BatchNorm (ablation) | LMD on SimpleBEV LiDAR-camera fusion · LiDAR + Camera modality replacement | Pearson correlation Rp/R: \textbf{0.08} $\pm$ 0.04 -> 0.41 $\pm$ 0.05 | Hurt |
Surprising findings
- The bias component alone produces non-trivial perception output, attributed mainly to linearized activation layers.
Limitations
- Attention modules introduce bilinear cross-modal terms that cannot be fully assigned to a single modality.
- Separation is validated only with perturbation-based proxy metrics rather than ground-truth attributions.
- Evaluated mainly on vehicle segmentation within SimpleBEV on nuScenes.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: Pearson correlation coefficient mean squared error
Baselines: SHAP LRP Baseline (unlinearized SimpleBEV)
Builds on: Layer-Wise Relevance Propagation Deep Taylor Decomposition SimpleBEV SHAP
Closest papers by meaning
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion (CVPR 2025 · similarity 0.87)
- AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking (NeurIPS 2024 · similarity 0.86)
- X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios (ICLR 2025 · similarity 0.85)
- GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.84)
- A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.84)
- Multiagent Multitraversal Multimodal Self-Driving: Open MARS Dataset (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.