Research map / Detection and segmentation / Open-set and few-shot detection
Multi-Scale Fusion for Object Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Scale Fusion (MSF), which builds an image pyramid, quantizes VAE representations per scale with shared/specific codebooks, and fuses super-pixels across scales to improve VAE guidance for transformer-based and diffusion-based object-centric learning.
Problem
Object-centric learning with VAE guidance uses single-scale VAE representations that are only high quality for certain object sizes, so objects of other sizes receive poor guidance.
Approach
Resize the input into three scales so all objects fall in the VAE's comfort zone, then fuse quantized VAE representations across scales (inter-scale fusion via a shared codebook selecting the most confident match; intra-scale fusion with scale-specific codebooks) and use the fused scale-1 representation as guidance.
Key results
- On ClevrTex, SLATE+MSF improves ARI-fg from 68.14 to 80.70 (about 12 points) over SLATE with random query initialization.
- On MOVi-D, STEVE+MSF raises ARI from 35.71 to 43.20 and mIoU from 19.10 to 21.21.
- On ClevrTex, SlotDiffusion+MSF improves ARI from 64.21 to 71.47, beating the GDR variant (69.21) and DINOSAUR (60.74) on ARI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Multi-Scale Fusion | SLATE_r · ClevrTex | ARI_fg: 68.14 -> 80.70 | Helped |
| add Multi-Scale Fusion | SLATE_r · COCO | ARI_fg: 24.54 -> 30.47 | Helped |
| add Multi-Scale Fusion | SLATE_r · VOC | ARI_fg: 15.65 -> 16.54 | Helped |
| add Multi-Scale Fusion | STEVE_c · MOVi-C | ARI: 52.20 -> 60.94 | Helped |
| add Multi-Scale Fusion | STEVE_c · MOVi-D | ARI: 35.71 -> 43.20 | Helped |
| add Multi-Scale Fusion | STEVE_c · MOVi-E | ARI: 28.00 -> 36.70 | Helped |
| add Multi-Scale Fusion | SlotDiffuz_r · VOC | ARI: 16.97 -> 19.40 | Helped |
| add intra-scale fusion (ablation) | MSF · ClevrTex | ARI+ARI_fg: 90.85 -> 100.71 | Helped |
| add inter-scale fusion (ablation) | MSF · ClevrTex | ARI+ARI_fg: 95.64 -> 100.71 | Helped |
| use scale-specific rather than shared scale-variant codebooks (ablation) | MSF · ClevrTex | ARI+ARI_fg: 92.28 -> 100.71 | Helped |
| use 3 scales rather than 2 (ablation) | MSF · ClevrTex | ARI+ARI_fg: 95.83 -> 100.71 | Helped |
| use 3 scales rather than 4 (ablation) | MSF · ClevrTex | ARI+ARI_fg: 98.96 -> 100.71 | Helped |
Surprising findings
- Inter-scale fusion alone can perform worse than disabling both fusion types (90.85 vs 91.03 ARI+ARI_fg), with intra-scale fusion needed to recover information.
- MSF gains on COCO and VOC are smaller than on synthetic datasets, attributed to objects often covering the whole scene beyond the coverage of the scales.
Limitations
- Gains are smaller on real-world datasets (COCO, VOC) where objects often span the whole scene beyond the chosen scales.
- Adds extra VAE pretraining computation and memory (about 31.25% more computation estimated).
- Scale count and resolution choices (N=3 at 128x128) are tuned empirically and may not transfer to all settings.
Experimental setup
Datasets: ClevrTex COCO VOC MOVi-C MOVi-D MOVi-E OpenImages subset
Benchmarks: ClevrTex COCO VOC MOVi-C MOVi-D MOVi-E OpenImages subset
Metrics: ARI ARI-fg mIoU mBO
Baselines: SLATE STEVE SlotDiffusion DINOSAUR SysBinder GDR
Builds on: SLATE STEVE SlotDiffusion GDR VQ-VAE Slot Attention
Closest papers by meaning
- Bootstrapping Top-down Information for Self-modulating Slot Attention (NeurIPS 2024 · similarity 0.85)
- Learning to Compose: Improving Object Centric Learning by Injecting Compositionality (ICLR 2024 · similarity 0.85)
- Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations (NeurIPS 2025 · similarity 0.85)
- Object-Centric Pretraining via Target Encoder Bootstrapping (ICLR 2025 · similarity 0.84)
- DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$ (ICLR 2024 · similarity 0.84)
- Multi-Part Object Representations via Graph Structures and Co-Part Discovery (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.