Research map / Architectures and efficiency / Attention and transformers
Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CACViT, a single pretrained plain ViT that performs feature extraction and exemplar matching jointly via self-attention for class-agnostic counting, with aspect-ratio-aware scale embedding and magnitude embedding.
Problem
Class-agnostic counting (CAC) estimates object counts in a query image given few exemplars; prior methods use an extract-then-match pipeline with separate feature extractors and post-matching, and resizing/softmax normalization lose scale and count-magnitude information.
Approach
Treat ViT self-attention as performing extraction (query-query and exemplar-exemplar blocks) and matching (query-exemplar blocks) simultaneously via concatenated query and exemplar tokens, then add an aspect-ratio-aware scale embedding for exemplars and a magnitude embedding that restores order-of-magnitude information in the similarity map.
Key results
- On FSC-147 3-shot, CACViT reaches 10.63 val MAE and 9.13 test MAE, a 19.04% and 23.60% relative MAE reduction over CounTR.
- On FSC-147 1-shot, CACViT reaches 11.41 val MAE and 8.62 test MAE, beating all prior 1-shot and 3-shot methods listed.
- On CARPK cross-dataset (no fine-tuning), CACViT reaches MAE 8.30, versus 10.44 for BMNet+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CACViT instead of CounTR in the 1-shot setting | FSC147 · FSC147 val, 1-shot | MAE: 13.15 -> 11.41; 13.23% | Helped |
| Use CACViT instead of CounTR in the 1-shot setting | FSC147 · FSC147 test, 1-shot | MAE: 12.06 -> 8.62; 28.52% | Helped |
| Use CACViT instead of CounTR in the 3-shot setting | FSC147 · FSC147 val, 3-shot | MAE: 13.13 -> 10.63; 19.04% | Helped |
| Use CACViT instead of CounTR in the 3-shot setting | FSC147 · FSC147 test, 3-shot | MAE: 11.95 -> 9.13; 23.60% | Helped |
| Use CACViT instead of BMNet+ without CARPK fine-tuning | CARPK counting · CARPK, not fine-tuned | MAE: 10.44 -> 8.30; 20.50% | Helped |
| Add the CLS attention component A_class (ablation) | CACViT · FSC147 test | MAE: 13.67 -> 11.56; 15.44% | Helped |
| Add aspect-ratio-aware scale embedding (SE) (ablation) | CACViT with CLS · FSC147 validation and test | MAE: 14.44% on the validation set; 5.80% on the test set | Helped |
| Add magnitude embedding (ME) without scale embedding (ablation) | CACViT with CLS · FSC147 validation and test | RMSE: 6.2; 7.35 | Helped |
| Add magnitude embedding (ME) without scale embedding (ablation) | CACViT with CLS · FSC147 validation and test | MAE | Mixed |
| Add magnitude embedding (ME) alongside scale embedding (SE) (ablation) | CACViT with CLS and SE · FSC147 test | MAE: 21.36% | Helped |
| Add magnitude embedding (ME) alongside scale embedding (SE) (ablation) | CACViT with CLS and SE · FSC147 test | RMSE: 41.05% | Helped |
Surprising findings
- The 1-shot setting outperforms the 3-shot setting on the FSC-147 test set, which the authors attribute to annotation quality in dense scenes.
Limitations
- Magnitude embedding gives marginal or negative MAE change on validation set, attributed to resized exemplars giving a misleading size prior.
- Evaluation limited to FSC147 and CARPK; code is only promised ('will be available').
Experimental setup
Datasets: FSC147 CARPK
Benchmarks: FSC147 CARPK
Metrics: MAE RMSE
Baselines: FamNet+ BMNet+ CounTR RCAC SAFECount SPDCN
Builds on: CounTR MAE BMNet+
Closest papers by meaning
- PBECount: Prompt-Before-Extract Paradigm for Class-Agnostic Counting (AAAI 2025 · similarity 0.92)
- VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting (AAAI 2024 · similarity 0.89)
- Point Segment and Count: A Generalized Framework for Object Counting (CVPR 2024 · similarity 0.88)
- Single Domain Generalization for Few-Shot Counting via Universal Representation Matching (CVPR 2025 · similarity 0.87)
- A Fixed-Point Approach to Unified Prompt-Based Counting (AAAI 2024 · similarity 0.87)
- DAVE - A Detect-and-Verify Paradigm for Low-Shot Counting (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.