Research map / Image generation / Autoregressive and latent generation
Tractable Transformers for Flexible Conditional Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tracformer, a sparse multi-scope encoder-decoder Transformer for non-autoregressive models that generalizes better to unseen conditional queries, beating SEDD, MDLM, BART and BERT on conditional perplexity.
Problem
Non-autoregressive generative models that excel at unconditional generation degrade sharply on conditional generation because they fail to generalize to conditional queries (unknown variable sets) unseen during training.
Approach
An encoder with multi-scope sparse attention whose layer-l scope grows exponentially (2^l), feeding a cross-attention-only decoder that routes local-to-global features, giving robustness to changes in the mask-token distribution while keeping global expressiveness and near-linear encoder cost.
Key results
- On zero-shot conditional perplexity with fixed masks on WikiText103, Tracformer reaches 29.38 vs. 30.36 for MDLM and 37.83 for SEDD (mask range [0.25,0.75]).
- On 1BW conditional perplexity with mask range [0.25,0.75], Tracformer achieves 41.98 vs. 55.19 (MDLM) and 53.72 (SEDD).
- Tracformer (109M) achieves 43.27 zero-shot unconditional PPL on WikiText103, behind MDLM (37.01) but ahead of SEDD on 1BW (51.34 vs. 79.29 for SEDD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Tracformer instead of the strongest listed diffusion baseline | zero-shot conditional generation · WikiText103, five of six fixed mask ranges | conditional perplexity | Helped |
| use Tracformer instead of the strongest listed diffusion baseline | zero-shot conditional generation · WikiText103, mask [0.2,0.3] & [0.4,0.6] | conditional perplexity: 24.31 vs 22.97 | Hurt |
| use Tracformer instead of the strongest listed diffusion baseline | zero-shot conditional generation · 1BW, six fixed mask ranges | conditional perplexity | Helped |
| use Tracformer instead of MDLM | zero-shot unconditional generation · WikiText103, WikiText2, Lambada, PTB, and 1BW | perplexity | Mixed |
| replace multi-scope attention with dense full attention (ablation) | Tracformer · WikiText103 CAR, Geometric(μ=50), mask ratio 0.1 | conditional perplexity: 445.46 -> 443.65 | No clear effect |
| use multi-scope attention instead of dense full attention (ablation) | Tracformer · WikiText103 CAR, Geometric(μ=50), mask ratio 0.9 | conditional perplexity: 574.73 -> 564.32 | Helped |
Surprising findings
- On WikiText103 with low mask ratios, MDLM slightly outperforms Tracformer, and strong unconditional PPL of diffusion models does not translate into conditional generation quality.
- On WikiText103 unconditional PPL, Tracformer is worse than MDLM and GPT-2 on several datasets such as LAMBADA and PTB.
Limitations
- Only trained at GPT-2 (base) scale due to resource limits.
- Does not consistently beat MDLM on conditional PPL at low mask ratios on WikiText103 and on some unconditional benchmarks (LAMBADA, PTB).
- Sparse attention lacks a custom kernel, so wall-clock efficiency gains are not realized yet.
Experimental setup
Datasets: WikiText103 LAMBADA 1BW OpenWebText WikiText2 PTB
Benchmarks: WikiText103 1BW LAMBADA WikiText2 PTB OpenWebText
Metrics: conditional perplexity unconditional perplexity MAUVE BERT Score BLEU-4
Baselines: BART BERT BERT-bidir SEDD MDLM GPT-2 D3PM PLAID
Builds on: BERT BART discrete diffusion language models rotary positional encoding
Closest papers by meaning
- TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings (AAAI 2025 · similarity 0.88)
- TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data (ICML 2025 · similarity 0.87)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
- A Theory for Conditional Generative Modeling on Multiple Data Sources (ICML 2025 · similarity 0.86)
- Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.86)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.