Research map / Vision-language and multimodal / Cross-modal learning
Free Lunch Enhancements for Multi-modal Crowd Counting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a 'free lunch' training strategy for multi-modal (RGB-T/RGB-D) crowd counting combining unsupervised post-pre-training cross-modal alignment (MoCo-style contrastive) and a regional density contrastive supervision loss, achieving state-of-the-art MAE below 10 on RGBT-CC without extra data or inference cost.
Problem
Multi-modal crowd counting suffers from a feature gap between modalities because backbones are pre-trained only on single-modal tasks, and point-level supervision lacks regional density context in congested scenes.
Approach
Insert an unsupervised cross-modal contrastive warm-up stage (PPCA) between single-modal pre-training and supervised fine-tuning, and add a regional density supervision loss that orders patch features by the density differences of their annotated patch counts, all using only existing training data.
Key results
- On RGBT-CC, GAME(0) reduced to 9.57 and RMSE to 17.05 versus BM's 10.19 and 17.32, the first MAE below 10 on RGBT-CC.
- On DroneRGBT, GAME(0) 4.94 and RMSE 7.81 versus BM's 6.20 and 10.40.
- Ablation on RGBT-CC: PPCA and RDS each reduce GAME(0) from 10.24 to 9.74 and 9.78 respectively in the three-branch setting; the full method reaches 9.57.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PPCA before fine-tuning (ablation) | BM with Swin-T, three branches · RGBT-CC | GAME(0): 10.24 -> 9.74 | Helped |
| Add RDS during fine-tuning (ablation) | BM with Swin-T, three branches and PPCA · RGBT-CC | GAME(0): 9.74 -> 9.57 | Helped |
| Add PPCA before fine-tuning (ablation) | Two-branch Swin-T baseline · RGBT-CC | GAME(0): 12.39 -> 10.02 | Helped |
| Add RDS to the two-branch model with PPCA (ablation) | Two-branch Swin-T with PPCA · RGBT-CC | GAME(0): 10.02 -> 9.89 | Helped |
| Add RDS without PPCA (ablation) | BM with Swin-T, three branches · RGBT-CC | GAME(0): 10.24 -> 9.78 | Helped |
| Add PPCA and RDS to BM (ablation) | BM with Swin-T · RGBT-CC | GAME(0): 10.24 -> 9.57 | Helped |
| Apply PPCA and RDS | BM with Swin-T · RGBT-CC | GAME(0): 10.19 -> 9.57 | Helped |
| Apply PPCA and RDS | BM with Swin-T · DroneRGBT | RMSE: 10.40 -> 7.81 | Helped |
| Apply PPCA and RDS | BM with Swin-T · ShanghaiTechRGBD | RMSE: 5.28 -> 5.38 | Hurt |
| Vary the RDS loss ratio (ablation) | Proposed method · RGBT-CC | GAME(0) | No clear effect |
| Set the RDS loss ratio to 0.1 (ablation) | Proposed method · RGBT-CC | GAME(0): GAME(0) is 9.57 | Helped |
Surprising findings
- Some methods that perform well on RGB-T datasets perform poorly on RGB-D (ShanghaiTechRGBD), where the proposed method's gain over BM is modest (GAME(0) 3.69 vs 3.72, RMSE 5.38 vs 5.28 worse).
Limitations
- Gains on ShanghaiTechRGBD over BM are marginal and mixed across metrics.
- Ablation gains per component are small (roughly 0.1-0.3 MAE).
Experimental setup
Datasets: RGBT-CC DroneRGBT ShanghaiTechRGBD ImageNet
Benchmarks: RGBT-CC DroneRGBT ShanghaiTechRGBD
Metrics: GAME(0) GAME(1) GAME(2) GAME(3) RMSE
Baselines: BM CSCA BL+IADM DEFNet MC3Net CAGNet VPMFNet BGDFNet
Builds on: BM (broker modality) MoCo DM-Count Swin Transformer
Closest papers by meaning
- Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting (CVPR 2021 · similarity 0.90)
- Counting without Scale: Scale-Consistent Error Correction for Crowd Counting (NeurIPS 2026 · similarity 0.88)
- Crowd Counting via Adversarial Cross-Scale Consistency Pursuit (CVPR 2018 · similarity 0.87)
- Leveraging Self-Supervision for Cross-Domain Crowd Counting (CVPR 2022 · similarity 0.87)
- Gramformer: Learning Crowd Counting via Graph-Modulated Transformer (AAAI 2024 · similarity 0.87)
- Taste More, Taste Better: Diverse Data and Strong Model Boost Semi-Supervised Crowd Counting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.