Research map / Detection and segmentation / Open-set and few-shot detection
GLID: Pre-training a Generalist Encoder-Decoder Vision Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLID, a generalist encoder-decoder vision model pre-trained with query-based masked image modeling on ImageNet-1K, then fine-tuned to many vision tasks by only swapping the final linear head.
Problem
Self-supervised pre-training (e.g., MAE) only transfers the encoder, so task-specific decoders must be trained from scratch on downstream tasks, creating a pretrain-finetune architecture gap and limiting benefits of large-scale pre-training.
Approach
Model both the pretext masked-image reconstruction and downstream tasks as query-to-answer problems with one Swin-based encoder plus query-based transformer decoder, pre-train the whole encoder-decoder, and fine-tune by replacing only the final linear head, initializing task queries from the pre-trained [CLS] token.
Key results
- GLID Swin-L reaches 78.5 APkp on COCO pose, 52.4 APbox on COCO detection, 53.9 mIoU on ADE20K, and 0.287 RMSE on NYUv2 depth, outperforming or matching specialists.
- Under the same Swin-B backbone and ImageNet-1K pre-training, GLID beats MAE, SimMIM, MixMAE, EsViT and supervised pre-training across all reported tasks (e.g., 51.2 mIoU vs 49.2 for MAE on ADE20K).
- With only 10% of ADE20K fine-tuning data, GLID reaches 31.2 mIoU vs 27.0 for supervised pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GLID pre-training instead of MixMAE pre-training | Swin-B · COCO | APkp: 75.8 -> 76.7 | Helped |
| use GLID pre-training instead of MAE pre-training | Swin-B · COCO | APbox: 49.2 -> 51.2 | Helped |
| use GLID pre-training instead of MAE pre-training | Swin-B · ADE20K | mIoU: 51.5 -> 52.7 | Helped |
| use GLID pre-training instead of MixMAE pre-training | Swin-B · ADE20K | mAP: 29.7 -> 30.9 | Helped |
| use GLID pre-training instead of MixMAE pre-training | Swin-B · ADE20K | PQ: 43.0 -> 44.7 | Helped |
| use GLID pre-training instead of SimMIM pre-training | Swin-B · NYUv2 | REL: 0.098 -> 0.085 | Helped |
| use GLID pre-training instead of MixMAE pre-training | Swin-B · NYUv2 | RMSE: 0.327 -> 0.293 | Helped |
| load all pre-trained encoder-decoder weights instead of backbone weights only (ablation) | GLID · ADE20K | mIoU: 50.9 -> 52.7 | Helped |
| load all pre-trained encoder-decoder weights instead of backbone weights only (ablation) | GLID · NYUv2 | RMSE: 0.363 -> 0.293 | Helped |
| use 3 decoder layers instead of 6 (ablation) | GLID · ADE20K | mIoU: 52.7 -> 50.7 | Hurt |
| use 9 decoder layers instead of 6 (ablation) | GLID · ADE20K | mIoU: 52.7 -> 52.7 | No clear effect |
| increase pre-training from 300 to 600 epochs (ablation) | GLID · ADE20K | mIoU: 51.5 -> 52.7 | Helped |
Surprising findings
- Multitask fine-tuning on ADE20K underperforms single-task fine-tuning (e.g., 51.8 mIoU vs 52.7 for semantic segmentation alone).
Limitations
- Multitask fine-tuning causes task interference and lowers performance versus single-task fine-tuning.
- Gains over specialists are modest on some tasks (e.g., depth and detection matched rather than clearly exceeded in some comparisons); comparisons use differing pre-training data across baselines.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K NYU-Depth V2
Benchmarks: COCO ADE20K NYU-Depth V2
Metrics: APkp APbox mIoU mAP PQ REL RMSE
Baselines: Mask2Former DETR Deformable-DETR ViTPose BinsFormer Pix2seq v2 UniT UViM Unified-IO Painter MAE SimMIM
Builds on: Masked Autoencoder (MAE) DETR Mask2Former AdaBins / BinsFormer ViTPose Swin Transformer
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.83)
- Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.83)
- Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering (ICML 2025 · similarity 0.83)
- Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning (CVPR 2025 · similarity 0.83)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.82)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.