Research map / Vision-language and multimodal / Human and embodied modeling
Non-autoregressive Sequence-to-Sequence Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NARVL, a non-autoregressive vision-language seq2seq model initialized from OFA that decodes all output tokens in parallel from learnable query tokens, trained with a Query-CTC loss and knowledge distillation, achieving comparable accuracy to AR OFA with 1.4-12.7x faster inference.
Problem
Autoregressive sequence-to-sequence vision-language models have high inference latency because they must run the decoder once per output token.
Approach
Replace the autoregressive decoder input with a fixed-length sequence of learnable query tokens decoded in one pass, and train it with a Query-CTC loss that marginalizes cross-entropy over all valid collapsed paths from the query positions to the target tokens, rather than over encoder-dependent alignments as in standard CTC.
Key results
- On RefCOCO val/testA/testB, NARVL-base reaches 88.78/90.63/84.67 vs AR OFA 88.15/90.08/83.45 while running 2.4x faster (54.8 ms vs 133.6 ms).
- On MSCOCO captioning, NARVL-KD-base reaches CIDEr 123.1 vs AR 138.2 with 7.2x speedup (51.2 ms vs 366.0 ms).
- On VQA test-dev, NARVL-KD-base scores 75.59 vs OFA-base beam search 77.48, with 50.76 ms vs 645.1 ms latency (12.7x speedup).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace autoregressive decoding with parallel decoding | OFA base on RefCOCO, RefCOCO+, and RefCOCOg · RefCOCO, RefCOCO+, RefCOCOg | Acc@0.5: RefCOCO: 88.15 -> 88.78 (Val), 90.08 -> 90.63 (TestA), 83.45 -> 84.67 (TestB); RefCOCO+: 81.67 -> 82.35 (Val), 86.40 -> | Helped |
| replace autoregressive decoding with parallel decoding | OFA base on SNLI-VE · SNLI-VE Dev and Test | accuracy: 89.0 -> 89.0 (Dev); 89.0 -> 89.0 (Test) | No clear effect |
| replace autoregressive decoding with parallel decoding | OFA base on VQA · VQA test-dev and test-std | accuracy: 77.48 -> 75.69 (test-dev); 77.58 -> 75.75 (test-std) | Hurt |
| replace autoregressive decoding with parallel decoding | OFA base on MSCOCO captioning · MSCOCO | BLEU@4: 41.0 -> 36.4 | Hurt |
| replace encoder-output decoder inputs with learnable query tokens (ablation) | NARVL on RefCOCO · RefCOCO | Acc@0.5: 87.78 -> 88.78 (Val); 89.89 -> 90.63 (TestA); 83.24 -> 84.67 (TestB) | Helped |
| replace encoder-output decoder inputs with learnable query tokens (ablation) | NARVL on MSCOCO captioning · MSCOCO | CIDEr: 121.8 -> 123.1 | Helped |
| replace cross-entropy loss with Q-CTC loss (ablation) | NARVL on MSCOCO captioning · MSCOCO | BLEU@4: 17.54 -> 26.69 (+9.15) | Helped |
| replace cross-entropy loss with Q-CTC loss (ablation) | NARVL on MSCOCO captioning · MSCOCO | CIDEr: 66.85 -> 93.24 (+26.39) | Helped |
| add knowledge distillation (ablation) | NARVL on VQA · VQA test-dev and test-std | accuracy: +1.48 (test-dev); +1.35 (test-std) | Helped |
| add knowledge distillation (ablation) | NARVL on MSCOCO captioning · MSCOCO | BLEU@4: 26.7 -> 36.4 (+9.7) | Helped |
| vary decoder query count from 10 to 20 (ablation) | NARVL on MSCOCO captioning · MSCOCO | BLEU@4: 29.43 -> 33.18 | Helped |
| vary decoder query count from 20 to 100, 500, or 1000 (ablation) | NARVL on MSCOCO captioning · MSCOCO | BLEU@4: 33.18 -> 32.47 (100); 32.36 (500); 31.91 (1000) | Hurt |
Surprising findings
- Query-CTC substantially outperforms cross-entropy on captioning (CIDEr 93.24 vs 66.85) when training the non-autoregressive decoder.
- Inference latency grows only modestly when increasing decoder query count from 10 to 1000 (51.32 ms to 60.40 ms) for captioning.
Limitations
- Optimization of NAR models is difficult; best VQA and captioning results rely on distillation from an autoregressive teacher, requiring two models.
- Conditional independence between output tokens can cause merged or incorrect output sequences.
- On captioning and VQA, NAR results remain below the AR counterpart (e.g., CIDEr 123.1 vs 138.2).
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg SNLI-VE VQA v2 MSCOCO Karpathy captions
Benchmarks: RefCOCO RefCOCO+ RefCOCOg SNLI-VE VQA v2 MSCOCO Karpathy test
Metrics: Acc@0.5 accuracy BLEU@4 METEOR CIDEr SPICE inference latency (ms)
Baselines: OFA MDETR UNICORN VL-T5 OSCAR SimVLM UNITER VILLA
Builds on: OFA CTC loss non-autoregressive transformer DETR
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.88)
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.88)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation (CVPR 2025 · similarity 0.87)
- Masked AutoDecoder is Effective Multi-Task Vision Generalist (CVPR 2024 · similarity 0.87)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.