Research map / Vision-language and multimodal / CLIP and zero-shot
VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLCounter, an end-to-end one-stage zero-shot object counter built on CLIP ViT-B/16 that avoids exemplar discovery, adding semantic-conditioned prompt tuning, learnable affine transformation, and segment-aware skip connections.
Problem
Zero-shot object counting (ZSOC) must count arbitrary named classes without exemplars, but existing two-stage pipelines (exemplar discovery then counting) suffer from error propagation and need extra training data.
Approach
Exploit CLIP's implicit patch-text similarity to localize targets in an end-to-end one-stage counter (VLBase), then tailor it with semantic-conditioned visual prompt tuning, a learnable affine transform on the similarity map, and skip connections from intermediate encoder layers gated by the counting map.
Key results
- On FSC147 test set, VLCounter reaches MAE 17.05 and RMSE 106.16, versus ZSC's 22.09 MAE (about 22.81% relative MAE improvement) and comparable to few-shot BMNet (16.71 MAE).
- Cross-dataset without fine-tuning, VLCounter achieves MAE 6.46 on CARPK and 48.94 on PUCPR+, improving over BMNet+ by up to 38.12% and 27.54% in MAE.
- Ablation on FSC147 test: SPT alone lowers MAE from 32.20 to 17.58; full model (SPT+LAT+SaSC) reaches 17.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SPT, LAT, and SaSC to VLBase (ablation) | VLBase on FSC147 test set · FSC147 test set | MAE: 32.20 -> 17.05 | Helped |
| Add SPT to VLBase (ablation) | VLBase on FSC147 test set · FSC147 test set | MAE: 32.20 -> 17.58 | Helped |
| Add LAT to VLBase (ablation) | VLBase on FSC147 test set · FSC147 test set | MAE: 32.20 -> 28.26 | Helped |
| Add SaSC to VLBase (ablation) | VLBase on FSC147 test set · FSC147 test set | MAE: 32.20 -> 24.16 | Helped |
| Add semantic conditioning to SPT (ablation) | VLCounter on FSC147 test set · FSC147 test set | MAE: 17.19 -> 17.05 | Helped |
| Add counting-map filtering to SaSC (ablation) | VLCounter on FSC147 test set · FSC147 test set | MAE: 19.38 -> 17.05 | Helped |
| Use plural instead of singular text prompts (ablation) | VLCounter on FSC147 test set · FSC147 test set | MAE: 19.18 -> 17.05 | Helped |
| Place prompt tokens in all image-encoder layers rather than layers 1–3 (ablation) | VLCounter on FSC147 test set · FSC147 test set | MAE: 19.73 -> 17.05 | Helped |
| Use VLCounter rather than VLBase | zero-shot counting on CARPK · CARPK | MAE: 20.47 -> 6.46 | Helped |
| Use VLCounter rather than VLBase | zero-shot counting on PUCPR+ · PUCPR+ | MAE: 90.82 -> 48.94 | Helped |
| Use VLCounter rather than ZSC | zero-shot counting on FSC147 test set · FSC147 test set | MAE: 22.09 -> 17.05 | Helped |
| Use VLCounter rather than BMNet+ | counting on FSC147 test set · FSC147 test set | MAE: 14.62 -> 17.05 | Mixed |
Surprising findings
- Using plural class-name prompts (e.g., 'a number of cats') instead of singular prompts improves counting MAE on FSC147 (17.05 vs 19.18 test MAE).
- Learnable affine parameters in LAT stayed close to their initialization (W near 1, B near 0), so the similarity map is only lightly adjusted.
Limitations
- Results on FSC147 still lag far behind few-shot methods on some metrics, e.g., RMSE on test set (106.16) is not best, and CLIP-Count has better val RMSE (61.18 vs 65.13).
- ZSC inference timing is not fully reported because its implementation is not publicized, so speed comparison is partial.
Experimental setup
Datasets: FSC147 CARPK PUCPR+
Benchmarks: FSC147 CARPK PUCPR+ IOCfish5k
Metrics: MAE RMSE inference speed learnable parameters MACs
Baselines: GMN FamNet BMNet BMNet+ RepRPN-Counter ZSC CLIP-Count
Builds on: CLIP visual prompt tuning class-agnostic counting (few-shot counters) rank-aware contrastive loss
Closest papers by meaning
- T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting (CVPR 2025 · similarity 0.92)
- Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting (AAAI 2024 · similarity 0.89)
- CountGD: Multi-Modal Open-World Counting (NeurIPS 2024 · similarity 0.89)
- Point Segment and Count: A Generalized Framework for Object Counting (CVPR 2024 · similarity 0.88)
- OmniCount: Multi-label Object Counting with Semantic-Geometric Priors (AAAI 2025 · similarity 0.88)
- Single Domain Generalization for Few-Shot Counting via Universal Representation Matching (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.