Research map / Vision-language and multimodal / Human and embodied modeling
CountGD: Multi-Modal Open-World Counting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Repurposes GroundingDINO for open-world object counting, letting users specify the target via text, visual exemplars, or both, and sets state-of-the-art on FSC-147 with multi-modal prompts.
Problem
Open-world object counting is limited: visual-exemplar counters are accurate but not flexible, while text-based counters are general but less accurate; no single model supports text, exemplars, or both.
Approach
Extends the open-vocabulary detector GroundingDINO by casting visual exemplars (RoIAlign features from the image backbone) as extra text tokens, fusing them with text via self-attention in the feature enhancer, and training only the new exemplar projection, feature enhancer and decoder while keeping the Swin and BERT encoders frozen; adds SAM-based test-time self-similarity normalization and adapt
Key results
- On FSC-147 test, CountGD with text and visual exemplars reaches MAE 6.75 (5.74 with corrections), beating the best visual-exemplar method DAVE (8.66) and all text-only methods.
- Text-only CountGD reaches test MAE 14.76 (12.98 corrected), comparable to GroundingREC (10.12) and better than CounTX (15.88) on RMSE-level metrics.
- On CountBench zero-shot with text only, CountGD obtains MAE 0.86 and RMSE 3.1 versus CounTX's 6.64 and 15.75.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add text prompts to visual-exemplar training and inference (ablation) | CountGD · FSC-147 test | MAE: 8.31 -> 5.74 | Helped |
| Add text prompts to visual-exemplar training and inference (ablation) | CountGD · FSC-147 test | RMSE: 91.05 -> 24.09 | Helped |
| Use visual exemplars instead of text alone (ablation) | CountGD · FSC-147 test | MAE: 12.98 -> 8.31 | Helped |
| Use visual exemplars instead of text alone (ablation) | CountGD · FSC-147 test | RMSE: 98.35 -> 91.05 | Helped |
| Add visual exemplars and text instead of text alone | CountGD · CARPK test | MAE: 3.83 -> 3.68 | Helped |
| Add visual exemplars and text instead of text alone | CountGD · CARPK test | RMSE: 5.41 -> 5.17 | Helped |
| Enable SAM TT-Norm (ablation) | CountGD · FSC-147 test | MAE: 10.92 -> 9.62 | Helped |
| Enable SAM TT-Norm (ablation) | CountGD · FSC-147 test | RMSE: 99.58 -> 98.90 | Helped |
| Enable adaptive cropping (ablation) | CountGD with SAM TT-Norm · FSC-147 test | RMSE: 98.90 -> 43.65 | Helped |
| Enable adaptive cropping (ablation) | CountGD with SAM TT-Norm · FSC-147 test | MAE: 9.62 -> 6.75 | Helped |
| Use text instead of incorrectly labeled visual exemplars for image 7171.jpg (ablation) | CountGD · FSC-147 test | RMSE: 43.65 -> 24.04 | Helped |
| Correct image 7611.jpg description to “yellow lego stud” (ablation) | CountGD · FSC-147 test | MAE: 5.70 -> 5.74 | No clear effect |
Surprising findings
- Visual-exemplar-only training of a GroundingDINO-based model reaches state-of-the-art MAE despite GroundingDINO being pretrained on text-to-image grounding rather than exemplars.
- Freezing the feature enhancer is needed for the text-exemplar interaction behaviours (filtering by color or location) to emerge.
Limitations
- No uncertainty or error estimates for predicted counts; the method relies on a confidence threshold.
- Output limited to 900 queries per pass, requiring adaptive cropping for very high counts.
- Some FSC-147 annotations were erroneous, requiring corrections that affect reported numbers.
Experimental setup
Datasets: FSC-147 FSC-147-D CARPK CountBench
Benchmarks: FSC-147 CARPK CountBench
Metrics: MAE RMSE
Baselines: CounTX CLIP-count VLCounter GroundingREC DAVE LOCA CounTR SAFECount Patch-selection DAVE_prm
Builds on: GroundingDINO CounTR Segment Anything Model DETR-style Hungarian matching
Closest papers by meaning
- OmniCount: Multi-label Object Counting with Semantic-Geometric Priors (AAAI 2025 · similarity 0.91)
- COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts (CVPR 2025 · similarity 0.89)
- VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting (AAAI 2024 · similarity 0.89)
- T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting (CVPR 2025 · similarity 0.89)
- Single Domain Generalization for Few-Shot Counting via Universal Representation Matching (CVPR 2025 · similarity 0.88)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.