Research map / Label-efficient and robust learning / Self-supervised and contrastive
Perceptual Group Tokenizer: Building Perception with Iterative Grouping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Perceptual Group Tokenizer (PGT), a visual backbone built entirely from iterative perceptual grouping operations (no self-attention), trained with DINO-style self-supervision, reaching 80.3% ImageNet-1K linear probe accuracy with adaptive inference-time group counts.
Problem
Whether perceptual grouping, rather than feature detection or self-attention, can serve as the driving principle of a visual recognition backbone that learns representations as strong as ViT/ConvNet under self-supervised learning.
Approach
Replace self-attention with a multi-head iterative grouping operator that binds input patch tokens to randomly sampled, learnable-distribution group tokens (fixed-point iterations with implicit differentiation), and use the group tokens as hypothesized contexts to refine patch features; the number of group tokens can be changed at inference without retraining.
Key results
- PGT-B achieves 80.3% top-1 ImageNet-1K linear probe accuracy (PGT-F-B-1024), matching DINO ViT-B/8 at 80.1%.
- With 12.5% of training group tokens at inference, PGT-B-256 keeps 72.1% top-1 accuracy; using more tokens at inference (1.5x) gives 79.9%.
- On ADE20k, PGT-B reaches 45.1 mIoU vs 44.1 for DINO + ViT-B/16 baseline; peak memory at 256 group tokens is 4.6% of ViT-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Multi-grouping (6 heads) instead of single grouping head (ablation) | PGT-Tiny · ImageNet-1K linear probe | top-1 accuracy: +4.1 (62.2 to 66.3) | Helped |
| Increasing group tokens at inference beyond training count (e.g., 256 trained, 384 inference) (ablation) | PGT-B and PGT-Tiny · ImageNet-1K linear probe | top-1 accuracy: 79.7 to 79.9 (PGT-B); 65.1 to 65.5 (Tiny) | Helped |
| Reducing group tokens at inference (256 trained to 16 inference) (ablation) | PGT-B-256 · ImageNet-1K linear probe | top-1 accuracy: 79.7 to 60.4 at 1/16 tokens | Hurt |
| Flat versus descending/ascending group token number across blocks (ablation) | PGT-Tiny · ImageNet-1K linear probe | top-1 accuracy: 63.1 flat vs 63.6 descend vs 63.7 flat in token shape row | No clear effect |
| Progressively increasing group token dimensions across blocks (ascending dims) (ablation) | PGT-Tiny · ImageNet-1K linear probe | top-1 accuracy: 63.4 ascend vs 62.0 descend vs 63.1 flat in token-size row | Mixed |
| Using 1024 group tokens at inference for segmentation (adaptive computation) | PGT-B · ADE20k semantic segmentation | mean IoU: +1.0 over DINO + ViT-B/16 (45.1 vs 44.1) | Helped |
| Increasing number of grouping iterations from 1 to 3 (ablation) | PGT-Tiny and PGT-B-256 · ImageNet-1K linear probe | top-1 accuracy: Tiny 61.4 to 65.1; B 79.3 to 79.7 | Helped |
| Peak memory of grouping vs ViT-B self-attention at 256 group tokens | PGT-B · inference peak memory | peak memory (% of ViT-B): 4.6% at 256 tokens vs 100% for ViT-B | Helped |
Surprising findings
- Increasing the number of group tokens at inference beyond the training count (out-of-distribution) can outperform the training setting, e.g. tiny model 65.5 vs 65.1 and PGT-B 79.9 vs 79.7.
- Conditional assignment entropy decreases during training while marginal entropy stays high, indicating no grouping collapse.
Limitations
- Relatively expensive computation due to iterative grouping processes.
- Grouping results sometimes produce fragmented parts rather than human-coherent objects.
- Gflops increase with group token count, exceeding ViT-B at 384+ tokens.
Experimental setup
Datasets: ImageNet-1K ADE20k
Benchmarks: ImageNet-1K linear probe ADE20k semantic segmentation
Metrics: top-1 accuracy mean IoU peak memory gflops entropy
Baselines: DINO ViT-B/16 DINO ViT-B/8 DINO ViT-S/8 iBOT BEiTv2 BYOL SwAV SCLR
Builds on: DINO ViT Slot Attention normalizing flows
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.85)
- Latent Noise Segmentation: How Neural Noise Leads to the Emergence of Segmentation and Grouping (ICML 2024 · similarity 0.85)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.85)
- GroupMamba: Efficient Group-Based Visual State Space Model (CVPR 2025 · similarity 0.84)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.84)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.