Research map / Image generation / Autoregressive and latent generation
DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DecQ, which adds 8 learnable detail-condensing queries that cross-attend to intermediate layers of a frozen DINOv2 encoder, feeds them to the decoder, and jointly denoises them with patch tokens, improving both reconstruction and generation in representation autoencoders.
Problem
Frozen vision foundation model (VFM) encoders in representation autoencoders (RAEs) under-represent low-level details such as color and texture, limiting reconstruction and fine-grained generation, while fine-tuning the VFM degrades its semantic latent space and generative quality.
Approach
Attach lightweight condenser modules (cross-attention plus FFN) to intermediate frozen VFM layers so learnable queries extract low-level details unidirectionally without altering patch tokens; the queries are fed to the decoder and jointly denoised with patch tokens during diffusion.
Key results
- DecQ raises reconstruction PSNR from 19.13 dB (RAE) to 22.76 dB with 8 queries and +3.9% FLOPs, and achieves the best rFID among VFM-based tokenizers (0.47).
- At 800 epochs, DecQ reaches gFID 1.41 without guidance and 1.05 with guidance on ImageNet 256x256.
- At 80 epochs, DecQ reaches gFID 1.80 without guidance versus 2.16 for RAE, reported as 3.3x faster convergence.
Surprising findings
- DecQ with the RAE decoder (discarding generated queries at inference) still outperforms RAE, suggesting query prediction helps the diffusion model generate better patch tokens.
- Fine-tuning the VFM yields far better reconstruction (PSNR 33.83) but much worse generation (gFID 57.81 at 80 epochs), confirming a reconstruction-generation trade-off.
Limitations
- Experiments limited to ImageNet at 256x256; no text-to-image or higher-resolution evaluation.
- Evaluated mainly with DINOv2 and SigLIP2 as VFMs; broader backbone coverage not studied.
- Additional queries and condensers add overhead that grows with more queries or denser placement.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256
Metrics: rFID PSNR SSIM gFID IS Precision Recall
Baselines: RAE SD-VAE VA-VAE SVG RPiAE FAE LV-RAE DMVAE REPA REG SiT MaskDiT
Builds on: Representation Autoencoders (RAE) LightningDiT DDT head REPA DINOv2
Closest papers by meaning
- Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.87)
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.86)
- DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence (NeurIPS 2026 · similarity 0.86)
- IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder (NeurIPS 2026 · similarity 0.86)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.85)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.