Research map / Image generation / Autoregressive and latent generation
SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates autoregressive sequence generation as imitation learning, minimizing occupancy-measure divergences (notably a chi-squared mixture divergence) non-adversarially, plus a backspace action, and finetunes Llama2-7b with it.
Problem
Maximum-likelihood training of autoregressive models does not constrain out-of-distribution behavior, so errors compound during generation, causing degenerate text and arithmetic mistakes.
Approach
Cast sequence modelling as an MDP with a backspace action and minimize occupancy-measure divergences (IQ-Learn-style, non-adversarial, single-variable Q/logit optimization), with a masking scheme that lets transformers train the backspace action in one pass; the chi-squared mixture divergence is recommended.
Key results
- On text generation with Llama-2-7b, SequenceMatch reaches MAUVE 0.91 ± 0.02 versus 0.85 ± 0.03 for MLE, with the best n-gram entropy (4.60).
- SequenceMatch's perplexity (7.13) is slightly worse than MLE (6.99), as expected since MLE optimizes log-perplexity directly.
- On arithmetic, SequenceMatch outperforms MLE and behavioral cloning across noise levels, and the model deletes OOD prefix tokens with backspaces to recover correct answers in three of four examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with SequenceMatch instead of MLE | Llama-2-7B · OpenWebText | MAUVE: 0.85 ± 0.03 -> 0.91 ± 0.02 | Helped |
| Train with SequenceMatch instead of MLE | Llama-2-7B · OpenWebText | n-gram entropy: 4.57 ± 0.02 -> 4.60 ± 0.01 | Helped |
| Train with SequenceMatch instead of MLE | Llama-2-7B · OpenWebText | Diversity: 0.56 ± 0.02 -> 0.56 ± 0.01 | No clear effect |
| Train with SequenceMatch instead of MLE | Llama-2-7B · OpenWebText | Perplexity: 6.99 ± 0.02 -> 7.13 ± 0.03 | Hurt |
| Add a <bkspc> action while training with MLE (ablation) | Llama-2-7B · OpenWebText | MAUVE: 0.85 ± 0.03 -> 0.84 ± 0.02 | Hurt |
| Add a <bkspc> action while training with MLE (ablation) | Llama-2-7B · OpenWebText | n-gram entropy: 4.57 ± 0.02 -> 4.59 ± 0.01 | Helped |
| Add a <bkspc> action while training with MLE (ablation) | Llama-2-7B · OpenWebText | Diversity: 0.56 ± 0.02 -> 0.56 ± 0.01 | No clear effect |
| Add a <bkspc> action while training with MLE (ablation) | Llama-2-7B · OpenWebText | Perplexity: 6.99 ± 0.02 -> 7.02 ± 0.02 | No clear effect |
| Train with MLE + ULK instead of MLE | Llama-2-7B · OpenWebText | MAUVE: 0.85 ± 0.03 -> 0.89 ± 0.02 | Helped |
| Add ground-truth noise demonstrations of <bkspc> use (ablation) | SequenceMatch · Arithmetic add-or-sub-in-base | accuracy | Helped |
| Add random noise demonstrations of <bkspc> use (ablation) | SequenceMatch · Arithmetic add-or-sub-in-base | accuracy | Helped |
| Use SequenceMatch loss instead of behavioral cloning or MLE | Llama-2-7B · Arithmetic add-or-sub-in-base | accuracy | Helped |
Surprising findings
- Adding a backspace action with MLE alone (MLE + backspace) lowers MAUVE to 0.84, below plain MLE.
Limitations
- Training overhead of 2-3x per gradient step versus MLE due to sampling and complex masks
- Perplexity is worse than MLE-trained models
- Experiments are limited to a single 7B model and small-scale arithmetic and openwebtext setups
Experimental setup
Datasets: math-dataset (add-or-sub-in-base) openwebtext
Benchmarks: math-dataset arithmetic held-out test (200 questions) openwebtext generation
Metrics: accuracy MAUVE n-gram entropy diversity perplexity
Baselines: MLE behavioral cloning MLE + contrastive sampling MLE + unlikelihood MLE + backspace
Builds on: IQ-Learn behavioral cloning unlikelihood training QLoRA
Closest papers by meaning
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.86)
- Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Bounds (NeurIPS 2026 · similarity 0.86)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.85)
- Architectural and Inferential Inductive Biases for Exchangeable Sequence Modeling (NeurIPS 2025 · similarity 0.85)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.85)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.