Research map / Architectures and efficiency / Attention and transformers
Masked AutoDecoder is Effective Multi-Task Vision Generalist
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Masked AutoDecoder (MAD), a multi-task vision generalist that decodes COCO vision task sequences (detection, instance segmentation, keypoints, captioning) in parallel with bi-directional attention and masked sequence modeling.
Problem
Autoregressive sequence-to-sequence Transformers used as vision generalists are poorly matched to vision task sequences, which lack strict sequential dependencies, and they are slow due to token-by-token decoding.
Approach
Replace uni-directional autoregressive decoding with parallel bi-directional decoding over task sequences, trained by randomly masking task-sequence tokens and reconstructing them (fully and partly masked sequences), with multi-stage masked inference refinement, all in one shared encoder-decoder with a single cross-entropy loss.
Key results
- MAD with ViT-B reaches 50.1 detection mAP, 41.2 instance segmentation mAP, 66.5 keypoint mAP, and 32.6 B@4 on COCO val, versus Pix2SeqV2's 46.5, 38.2, 64.8, 34.9.
- MAD is about 157x faster than Pix2SeqV2 in total inference time (0.26 s vs 40.86 s with ViT-B backbone per the table).
- Ablation: parallel decoding cuts inference time from 3953 ms to 137 ms, with masked training and masked inference raising detection from 27.9 to 38.9 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace autoregressive decoding with parallel decoding and bidirectional attention (ablation) | MAD · COCO | object detection mAP: 27.9 -> 35.9 | Helped |
| add masked sequence training (ablation) | MAD with parallel decoding · COCO | object detection mAP: 35.9 -> 38.9 | Helped |
| add masked inference (ablation) | MAD with masked training · COCO | image captioning B@4: 18.6 -> 29.6 | Helped |
| add masked inference (ablation) | MAD with masked training · COCO | object detection mAP: 38.9 -> 38.9 | No clear effect |
| use a single training mask ratio instead of random or multiple ratios (ablation) | MAD · COCO | instance segmentation mAP: 31.9, 32.0, 32.3 | No clear effect |
| vary the training mask ratio from 0.4 to 0.8 (ablation) | MAD · COCO | image captioning B@4: 29.2; 29.7; 29.6; 28.4 | Mixed |
| increase coordinate quantization bins from 300 to 500 (ablation) | MAD · COCO object detection | mAP: 38.5 -> 38.9 | Helped |
| increase coordinate quantization bins beyond 500 (ablation) | MAD · COCO object detection | mAP: 38.9 -> 38.6 | Hurt |
| increase bit-mask size from 12 to 16 (ablation) | MAD · COCO instance segmentation | mAP: 31.5 -> 32.3 | Helped |
| use progressively decreasing inference mask ratios {0.8, 0.6, 0.4} instead of no masked inference (ablation) | MAD · COCO image captioning | B@4: 18.6 -> 29.6 | Helped |
| use joint training with masked auto-decoding rather than joint training without it (ablation) | MAD · COCO | object detection mAP: 35.9 -> 38.9 | Helped |
| use MAD instead of Pix2SeqV2 | ViT-B vision generalist · COCO val object detection | mAP: 46.5 -> 50.1 | Helped |
Surprising findings
- Autoregressive decoding is better than parallel decoding alone on image captioning, and masked inference is needed to close the gap (29.6 vs 18.2 B@4 without masking).
Limitations
- Evaluated only on four tasks on COCO, so generality as a vision generalist is untested.
- Captioning still lags behind autoregressive models in language modeling context.
- Inference speed of masked refinement adds overhead relative to single-pass parallel decoding.
Experimental setup
Datasets: MS-COCO
Benchmarks: MS-COCO val
Metrics: mAP BLEU@4 inference time
Baselines: Pix2SeqV2 VisionLLM Faster R-CNN DETR Pix2Seq Mask R-CNN Keypoint R-CNN Transformer captioning
Builds on: Pix2SeqV2 DETR BERT Hungarian matching masked image modeling
Closest papers by meaning
- Non-autoregressive Sequence-to-Sequence Vision-Language Models (CVPR 2024 · similarity 0.87)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.85)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.85)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.85)
- Each Complexity Deserves a Pruning Policy (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.