Research map / Detection and segmentation / Open-set and few-shot detection
Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Causal Mode Multiplexer (CMM), which switches between total-effect and total-indirect-effect learning based on ROTO/RXTO/ROTX input type to remove thermal direct-effect bias in RGBT pedestrian detection, and releases the ROTX-MP test set of 1000 image pairs.
Problem
Multispectral pedestrian detectors learn statistical co-occurrence between pedestrian labels and thermal features from day/night datasets, so they fail on ROTX scenes where pedestrians are visible in RGB but thermally obscured.
Approach
Learns causal mode interchangeably: total effect for daytime (ROTO) inputs and total indirect effect (thermal direct effect pruned via counterfactual intervention) for nighttime/obscured inputs, selected automatically by a Gumbel-softmax-based Causal Mode Multiplexing loss and switchable TIE at inference.
Key results
- On ROTX-MP trained on KAIST, CMM reaches 70.44 AP versus 21.69 for the uncertainty-guided baseline and 36.95 for HFF.
- On existing benchmarks CMM improves over its baseline: KAIST MR 8.54 vs 8.67, CVC-14 MR 17.13 vs 18.70, FLIR AP 87.80 vs 84.67.
- Trained on CVC-14 and FLIR, CMM achieves 34.96 and 57.09 AP on ROTX-MP, far above the baseline's 13.36 and 12.23.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CMM with switchable total indirect effect | Uncertainty-Guided model · KAIST All | MR: 8.67 -> 8.54 | Helped |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model · CVC-14 Night | MR: 11.08 -> 7.71 | Helped |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model · CVC-14 Day | MR: 23.87 -> 27.81 | Hurt |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model · FLIR All | AP: 84.67 -> 87.80 | Helped |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model trained on KAIST · ROTX-MP All | AP: 21.69 -> 70.44 | Helped |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model trained on CVC-14 · ROTX-MP All | AP: 13.36 -> 34.96 | Helped |
| apply CMM with switchable total indirect effect | Uncertainty-Guided model trained on FLIR · ROTX-MP All | AP: 12.23 -> 57.09 | Helped |
| use TE for training and TIE for inference instead of switchable TE/TIE (ablation) | CMM · ROTX-MP All (train KAIST) | AP: 70.44 -> 57.05 | Hurt |
| use TE for training and TIE for inference instead of switchable TE/TIE (ablation) | CMM · KAIST All (train KAIST) | MR: 8.54 -> 27.27 | Hurt |
| use TIE for training and inference instead of switchable TE/TIE (ablation) | CMM · ROTX-MP All (train KAIST) | AP: 70.44 -> 56.45 | Hurt |
| use TIE for training and inference instead of switchable TE/TIE (ablation) | CMM · KAIST All (train KAIST) | MR: 8.54 -> 12.10 | Hurt |
| use TE for training and TIE for inference instead of switchable TE/TIE (ablation) | CMM · ROTX-MP All (train CVC-14) | AP: 34.96 -> 27.97 | Hurt |
Surprising findings
- Standard multispectral detectors achieve only 3.64 AP (CFT) on ROTX-MP when trained on KAIST, showing near-total failure on thermally obscured pedestrians.
Limitations
- ROTX-MP is small (1000 test images only, no training split) and was hand-picked from videos of 8 volunteers.
- Some ROTX-MP scenarios are narrow (glass windows, heat-insulation clothing), limiting coverage of obscured-thermal conditions.
- CMM slightly degrades CVC-14 day MR (27.81 vs 23.87 baseline is worse), indicating a trade-off on some splits.
Experimental setup
Datasets: KAIST CVC-14 FLIR
Benchmarks: KAIST CVC-14 FLIR ROTX-MP
Metrics: Log Average Miss Rate (MR) Average Precision (AP)
Baselines: Kim et al. (uncertainty-guided) CFT MBNet HFF (Halfway Fusion + Faster R-CNN) TE+TIE TIE+TIE
Builds on: Uncertainty-Guided multispectral pedestrian detection (Kim et al.) total indirect effect debiasing Gumbel-softmax
Closest papers by meaning
- Robust Pedestrian Detection with Uncertain Modality (AAAI 2026 · similarity 0.87)
- Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory (AAAI 2022 · similarity 0.86)
- Multispectral Pedestrian Detection with Sparsely Annotated Label (AAAI 2025 · similarity 0.84)
- DiMSOD: A Diffusion-Based Framework for Multi-Modal Salient Object Detection (AAAI 2025 · similarity 0.82)
- Unbiased Faster R-CNN for Single-source Domain Generalized Object Detection (CVPR 2024 · similarity 0.82)
- MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.