Research map / Image generation / Autoregressive and latent generation

JetFormer: An autoregressive generative model of raw images and text

ICLR 2025 · Image generation › Autoregressive and latent generation · approach: Next-scale and token autoregressive images · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes JetFormer, a decoder-only transformer trained end-to-end from raw pixels and text, using a normalizing flow to produce soft-token image representations that feed a GMM-headed autoregressive model.

Problem

Joint generative modeling of images and text typically relies on separately pretrained, frozen, lossy image tokenizers (VQ-VAE/VAE), which limits end-to-end optimization and likelihood-based training on raw pixels.

Approach

Use an invertible normalizing flow as a trainable image encoder/decoder producing soft tokens, trained jointly with an autoregressive transformer via a GMM loss on image soft tokens and cross-entropy on text, with an RGB Gaussian-noise curriculum and factored-out redundant dimensions to guide learning toward high-level structure.

Key results

• JetFormer-L achieves FID 6.64 on ImageNet256 class-conditional generation (500 epochs) without any pretrained component, competitive with GIVT-Causal (3.35) and LDM-4-G (3.60).
• JetFormer-L (T2I) reaches zero-shot COCO FID-30k of 18.63 at 2.75B parameters, versus 12.08 for MAGVLT (T&I) and 10.39 for DALL-E-2.
• JetFormer-L (T&I) reaches COCO captioning CIDEr 119.8 and VQAv2 accuracy 70.0 after fine-tuning, exceeding CapPa and CLIP L/14 at the 900M-pair setting.

Measured effects reported in the paper

Surprising findings

• Removing the noise curriculum raises ImageNet FID from 7.84 to 44.71 while NLL barely changes (3.14 vs 3.05), showing NLL alone does not track sample quality.
• A PCA-preprocessed variant gets similar FID with the noise curriculum less important, yet end-to-end JetFormer still performs better in ablations.

Limitations

• Sample visual quality lags behind state-of-the-art diffusion models that use pretrained latent representations.
• Full end-to-end training increases computational requirements.
• Text-to-image quality is worse than VQ-VAE-based models like MAGVLT, while the T&I variant trades off T2I quality for understanding.

Experimental setup

Datasets: ImageNet1k WebLI C4 (tokenizer) MS-COCO VQAv2

Benchmarks: ImageNet256 class-conditional generation (ADM FID-50k) MS-COCO FID-30k zero-shot and fine-tuned ImageNet zero-shot classification COCO captioning CIDEr VQAv2 test-dev

Metrics: FID precision recall NLL (bits per dimension) CIDEr VQAv2 accuracy zero-shot ImageNet accuracy

Baselines: BigGAN-deep ADM-G LDM-4-G VQGAN ViT-VQGAN GIVT-Causal DALL-E CogView CogView2 MAGVLT ARGVLT Make-A-Scene

Builds on: GIVT Gemma normalizing flow coupling layers (Glow-style) VQGAN classifier-free guidance

Closest papers by meaning

• STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation (NeurIPS 2026 · similarity 0.89)
• JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)
• Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.88)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
• A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation (ICLR 2025 · similarity 0.88)