Research map / Detection and segmentation / CNN features
SalM²: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SalM², a 0.08M-parameter Mamba-based dual-branch driver attention predictor that fuses CLIP-derived driving-scene semantics with bottom-up image features via cross-modal channel attention.
Problem
Driver attention prediction in driving scenes is typically done with large, complex models that rely on visual features and ignore task-relevant semantic information of the scene.
Approach
A lightweight Mamba-based backbone with a Selective Channel Parallel Mamba (SCPM) layer that splits channels into parallel Mamba branches to avoid parameter explosion, plus a cross-modal attention (CMA) module that injects CLIP semantic features into the deepest image features using channel attention with a single learnable scale.
Key results
- SalM² reaches 0.08M parameters and 4.45 GFLOPs on BDD-A, with CC 0.64, SIM 0.47 and KLD 1.08, matching or nearly matching the best of the compared models.
- On TrafficGaze, SalM² achieves best AUC_Judd (0.98), CC 0.94 and KLD 0.28, while using a fraction of the parameters of competitors.
- On DrFixD-rainy, SalM² achieves best CC (0.86), NSS (4.31) and KLD (0.47), with AUC_Judd 0.95 and SIM 0.68 (second-best).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Wu et al. backbone with the SalM² backbone without semantic guidance (ablation) | driver attention prediction backbone · TrafficGaze | NSS: 5.82 -> 5.87 | Helped |
| Replace the Wu et al. backbone with the SalM² backbone without semantic guidance (ablation) | driver attention prediction backbone · DrFixD-rainy | AUC_Borji: 0.90 -> 0.89 | Hurt |
| Replace the Wu et al. backbone with the SalM² backbone without semantic guidance (ablation) | driver attention prediction backbone · DrFixD-rainy | NSS: 4.21 -> 4.29 | Helped |
| Replace the Wu et al. backbone with the SalM² backbone without semantic guidance (ablation) | driver attention prediction backbone · BDDA | CC: 0.62 -> 0.63 | Helped |
| Increase input resolution from 256² to 512² (ablation) | SalM² · TrafficGaze | CC: 0.94 -> 0.95 | Helped |
| Increase input resolution from 256² to 512² (ablation) | SalM² · DrFixD-rainy | NSS: 4.31 -> 4.26 | Hurt |
| Increase input resolution from 256² to 512² (ablation) | SalM² · BDDA | CC: 0.64 -> 0.64 | No clear effect |
| Add driving-scene semantic guidance (ablation) | SalM² backbone · driver-attention scenes with a car crossing or pedestrian ready to cross | attention allocation to task-relevant object | Helped |
Surprising findings
- Removing the semantic branch yields a backbone that performs similarly or better on some metrics (e.g., the backbone alone beats the full model on TrafficGaze CC/SIM/KLD at 256 resolution in the backbone ablation), suggesting limited semantic contribution on some datasets.
- Increasing input resolution from 256 to 512 did not improve BDD-A performance, which the authors attribute to a mismatch between the bottom-up and CLIP top-down branches.
Limitations
- CLIP semantic branch limited to 224x224 input, so higher resolution does not add semantic information
- Semantic gains are dataset-dependent; on some datasets the backbone alone is competitive
- Evaluation only on three driver-attention datasets and compared against a fixed set of baselines; SCOUT was run without its extra annotations
Experimental setup
Datasets: TrafficGaze DrFixD-rainy BDD-A
Benchmarks: BDD-A TrafficGaze DrFixD-rainy
Metrics: AUC_Borji AUC_Judd NSS CC SIM KLD
Baselines: MLNet CDNN FBLNet SCOUT TASED-Net DrFixD-rainy SCAFNet
Builds on: Mamba CLIP Wu et al. Mamba-based segmentation network
Closest papers by meaning
- Samba: A Unified Mamba-based Framework for General Salient Object Detection (CVPR 2025 · similarity 0.86)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.85)
- MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception (CVPR 2025 · similarity 0.84)
- H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving (AAAI 2025 · similarity 0.84)
- MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations (ICLR 2025 · similarity 0.84)
- Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSM (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.