Research map / Detection and segmentation / Open-set and few-shot detection
SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SPOT improves slot-based autoencoders for unsupervised object-centric learning via attention-mask self-distillation from decoder to slot-attention encoder plus patch-order permutations in the autoregressive transformer decoder.
Problem
Unsupervised object-centric learning with slot-based autoencoders struggles on complex real-world images; the encoder's slot generation is weak and autoregressive transformer decoders tend to ignore slot vectors under teacher forcing.
Approach
Distill decoder cross-attention masks (better object masks) into the encoder's slot-attention masks through a two-stage teacher-student Hungarian-matched cross-entropy loss, and randomly permute the patch prediction order (with per-permutation BOS tokens) in the autoregressive decoder so later tokens must rely on slots.
Key results
- On COCO, SPOT reaches 34.7 MBOi and 44.3 MBOc without ensembling (35.0 / 44.7 with test-time ensembling), beating prior SOTA by 2.7 MBOi and 5.9 MBOc points.
- Ablation on COCO: sequence permutations without self-training improve decoder MBOi from 32.0 to 32.7; with self-training, the full combination improves decoder MBOi from 33.2 to 34.7.
- Self-training with an MLP decoder on COCO raises MBOi from 26.7 to 28.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sequence permutations to the autoregressive decoder (ablation) | SPOT · COCO | MBOi: 30.1±0.5 -> 33.7±0.1 | Helped |
| add self-training by distilling decoder attention masks to slot attention (ablation) | SPOT with sequence permutations · COCO | MBOi: 32.7±0.2 -> 34.7±0.1 | Helped |
| ensemble nine permutations at test time (ablation) | SPOT · COCO | MBOi: 34.7±0.1 -> 35.0±0.1 | Helped |
| use parallel prediction for 25% of training iterations instead of sequence permutations (ablation) | autoregressive transformer decoder · COCO | MBOi: 32.7 -> 27.8 | Hurt |
| add self-training (ablation) | SPOT with an MLP decoder · COCO | MBOi: 26.7 -> 28.4 | Helped |
| fine-tune the image encoder during the self-training stage (ablation) | SPOT · COCO | MBOi: 34.7±0.1 -> 32.4±0.2 | Hurt |
| set self-training loss weight λ to 0 (ablation) | SPOT · COCO | MBOi: 34.7±0.1 -> 30.7±2.2 | Hurt |
| use default rather than random permutation to generate teacher target masks (ablation) | SPOT self-training · COCO | MBOi: 34.6±0.2 -> 34.7±0.1 | No clear effect |
| add self-training without sequence permutations (ablation) | SPOT · COCO | MBOi: 32.0±0.1 -> 30.5±0.2 | Mixed |
Surprising findings
- Without sequence permutations, self-training boosts slot-attention masks but hurts decoder-mask performance, indicating the autoregressive decoder begins to ignore slots.
- Parallel (non-autoregressive) decoding during training, as in CapPa, does not help object-centric learning, unlike sequence permutations.
Limitations
- Test-time ensembling of permutations increases inference time for a small gain.
- FG-ARI is not a primary metric and may be misleading; results are focused on synthetic and COCO/PASCAL benchmarks.
Experimental setup
Datasets: MS COCO 2017 PASCAL VOC 2012 MOVi-C MOVi-E
Benchmarks: MS COCO 2017 PASCAL VOC 2012 MOVi-C MOVi-E
Metrics: MBOi MBOc MIOU FG-ARI
Baselines: SA SLASH SLATE CAE DINOSAUR DINOSAUR-MLP Rotating Features SlotDiffusion (Stable-)LSD
Builds on: Slot Attention DINOSAUR SLATE DINO Bi-level optimization for slot attention
Closest papers by meaning
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.89)
- MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning (NeurIPS 2025 · similarity 0.88)
- Grounded Object-Centric Learning (ICLR 2024 · similarity 0.88)
- Object-Centric Pretraining via Target Encoder Bootstrapping (ICLR 2025 · similarity 0.88)
- GLASS: Guided Latent Slot Diffusion for Object-Centric Learning (CVPR 2025 · similarity 0.87)
- Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.