Research map / Image generation / Autoregressive and latent generation
Seg-VAR:Image Segmentation with Visual Autoregressive Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seg-VAR reframes image segmentation as conditional visual autoregressive mask generation, encoding masks as spatially-aware "seglat" latent tokens and generating them with a VAR-style model conditioned on images.
Problem
Visual autoregressive modeling has been used for image generation, but its potential for precise low-level spatial segmentation has not been explored.
Approach
Rethinks segmentation as conditional autoregressive mask generation: masks are encoded into a multi-scale latent language ("seglat") with location-sensitive color mapping, and a VAR-style transformer generates these tokens; the seglat encoder and decoder are jointly trained rather than frozen.
Key results
- On ADE20K, Seg-VAR reaches 43.2 AP and 54.9 mIoU versus 36.3 AP and 48.5 mIoU for GSS.
- On Cityscapes, Seg-VAR reaches 49.5 AP, 85.8 mIoU and 66.8 PQ versus 43.1 AP, 80.1 mIoU and 59.3 PQ for GSS.
- Image FID at depth 30 is 1.97 for Seg-VAR, matching VAR and slightly better than ControlVAR (1.98).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Seg-VAR instead of GSS | Image segmentation · ADE20K | AP: 36.3 -> 43.2 | Helped |
| Use Seg-VAR instead of GSS | Image segmentation · ADE20K | mIoU: 48.5 -> 54.9 | Helped |
| Use Seg-VAR instead of GSS | Image segmentation · CityScapes | AP: 43.1 -> 49.5 | Helped |
| Use Seg-VAR instead of GSS | Image segmentation · CityScapes | mIoU: 80.1 -> 85.8 | Helped |
| Use Seg-VAR instead of GSS | Image segmentation · CityScapes | PQ: 59.3 -> 66.8 | Helped |
| Add joint training of the seglat encoder and decoder instead of the UU design (ablation) | Seg-VAR · ADE20K | AP: 38.6 -> 43.2 | Helped |
| Add joint training of the seglat encoder and decoder instead of the UU design (ablation) | Seg-VAR · ADE20K | mIoU: 50.4 -> 54.9 | Helped |
| Add joint training of the seglat encoder and decoder instead of the UU design (ablation) | Seg-VAR · CityScapes | AP: 44.8 -> 49.5 | Helped |
| Add joint training of the seglat encoder and decoder instead of the UU design (ablation) | Seg-VAR · CityScapes | mIoU: 81.3 -> 85.8 | Helped |
| Add control mechanisms to VAR | Image generation, depth 16 · Image generation, depth 16 | FID: 3.60 -> 3.8 | Hurt |
| Add control mechanisms to VAR | Image generation, depth 20 · Image generation, depth 20 | FID: 2.95 -> 3.05 | Hurt |
| Add control mechanisms to VAR | Image generation, depth 24 · Image generation, depth 24 | FID: 2.33 -> 2.56 | Hurt |
Surprising findings
- Image-control generation FID of Seg-VAR is slightly worse than VAR at smaller depths (e.g., 3.8 vs 3.60 at depth 16), attributed to added control complexity.
Limitations
- Image FID slightly worse than VAR at smaller model depths due to control integration overhead.
Experimental setup
Datasets: COCO ADE20K Cityscapes
Benchmarks: COCO val2017 ADE20K Cityscapes
Metrics: AP mIoU PQ FID FLOPs fps
Baselines: GSS VAR ControlVAR Mask2Former Mask R-CNN
Builds on: VAR ControlVAR Mask2Former Detectron2
Closest papers by meaning
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning (AAAI 2024 · similarity 0.87)
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.87)
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model (NeurIPS 2025 · similarity 0.86)
- EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation (AAAI 2025 · similarity 0.86)
- Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.