Research map / Detection and segmentation / Open-set and few-shot detection
Point Segment and Count: A Generalized Framework for Object Counting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PseCo, a point-segment-count framework that combines a keypoint-based point decoder, SAM mask proposals, and CLIP-embedding classifiers trained with hierarchical distillation for few-shot and zero-shot object counting and detection.
Problem
Class-agnostic object counting (few-shot with exemplar boxes, zero-shot with class names) where density-based methods lack interpretability and naive SAM+CLIP is slow and misses small crowded objects.
Approach
Replace uniform grid prompts for SAM with a learned class-agnostic keypoint heatmap that yields few accurate point prompts, then classify SAM's hierarchical mask proposals with CLIP text/image embeddings as classifier weights, trained with a hierarchical knowledge distillation loss from CLIP.
Key results
- Few-shot counting on FSC-147 test: MAE 13.05 and RMSE 112.86, comparable to CounTR (11.95 MAE) and BMNet+ (14.62 MAE) and better than C-DETR (16.79 MAE).
- Few-shot detection on FSC-147 reaches AP 43.53 vs 22.66 for C-DETR.
- On COCO, novel-class AP50n reaches 32.9 vs 27.8 for Detic; on LVIS, novel APm reaches 20.6 vs 17.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace grid-point prompts with class-agnostic heatmap localization (ablation) | PseCo · FSC-147 test | AP: 41.66 -> 43.53 | Helped |
| include SAM-detected objects alongside ground-truth points in the localization target (ablation) | PseCo point decoder · FSC-147 test | AP: 43.26 -> 43.53 | Helped |
| use SAM ViT-H instead of SAM ViT-B (ablation) | PseCo · FSC-147 test | AP: 39.83 -> 43.53 | Helped |
| use SAM ViT-H instead of SAM ViT-L (ablation) | PseCo · FSC-147 test | AP: 42.58 -> 43.53 | Helped |
| add hierarchical knowledge distillation loss Lkd (ablation) | PseCo classifier · FSC-147 test | AP: 41.69 -> 43.53 | Helped |
| add hierarchical knowledge distillation loss Lkd (ablation) | PseCo classifier · FSC-147 test | RMSE: 127.95 -> 112.86 | Helped |
| use class-agnostic localization with ViLD rather than SAM grid prompts | ViLD zero-shot counting · FSC-147 test | MAE: 42.48 -> 17.80 | Helped |
| replace ViLD classification with the PseCo classifier | class-agnostic localization zero-shot counting · FSC-147 test | MAE: 17.80 -> 16.58 | Helped |
| use PseCo instead of Detic | open-vocabulary detection · COCO novel classes | AP50: 27.8 -> 32.9 | Helped |
| use PseCo instead of Detic | open-vocabulary detection · LVIS novel classes | mask AP: 17.8 -> 20.6 | Helped |
Surprising findings
- Zero-shot counting lags few-shot counting substantially on FSC-147 (test MAE 16.58 vs 13.05), attributed to ambiguous class names such as 'go game' for go pieces.
- PseCo performs better zero-shot than few-shot on FSCD-LVIS but the reverse on FSC-147, which the authors attribute to poor example images in FSCD-LVIS.
Limitations
- Not robust to occluded objects because SAM cannot distinguish occluded instances.
- Zero-shot performance on FSC-147 lags few-shot, possibly due to ambiguous class names.
- Pseudo box labels generated from ground-truth points are noisy for training the classifier.
Experimental setup
Datasets: FSC-147 FSCD-LVIS COCO LVIS
Benchmarks: FSC-147 FSCD-LVIS COCO LVIS
Metrics: MAE RMSE AP AP50 APm AP50n
Baselines: C-DETR CounTR BMNet+ FamNet SAM-Free ZSOC ViLD Detic
Builds on: SAM CLIP ViLD Detic Faster R-CNN
Closest papers by meaning
- A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation (NeurIPS 2024 · similarity 0.91)
- DAVE - A Detect-and-Verify Paradigm for Low-Shot Counting (CVPR 2024 · similarity 0.89)
- VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting (AAAI 2024 · similarity 0.88)
- Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting (AAAI 2024 · similarity 0.88)
- Learning to Count without Annotations (CVPR 2024 · similarity 0.87)
- Single Domain Generalization for Few-Shot Counting via Universal Representation Matching (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.