Research map / Image generation / Fast diffusion sampling
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TS-DFM, which replaces blind stochastic CTMC jumps at RK-4 midpoints in few-step discrete flow matching distillation with energy-compass-guided sequence-to-token navigation, yielding better few-step perplexity than FS-DFM and the teacher at training-only cost.
Problem
Few-step distillation of discrete flow matching language models underperforms; the paper argues the bottleneck is the low-quality training trajectories built from blind stochastic jumps at RK-4 midpoints, not student capacity.
Approach
Generate K candidate CTMC continuations at each navigated midpoint, select the lowest-energy one with a frozen, time-agnostic energy compass trained on generation-aware negatives (velocity near-miss), then refine tokens using velocity confidence with a safeguard; navigation activates only for t >= tau.
Key results
- At 8 steps with uniform source, TS-DFM achieves 56.1 GPT-2 PPL vs 87.6 for FS-DFM (36% reduction) and 82.8 for the 1024-step teacher.
- For the mask source at 8 steps, TS-DFM (DFM init) reaches 91.4 PPL vs 516.6 for FS-DFM (5.6x improvement), beating the 1024-step teacher's 93.6.
- At 1.3B parameters, TS-DFM at 8 steps reaches 48.0 GPT-2 Large PPL vs 57.5 for the teacher and 81.5 for FS-DFM.
Surprising findings
- A 8-step student trained with shaping beats the 1024-step teacher on uniform and mask sources, suggesting trajectory quality rather than student capacity limits few-step distillation.
- Lower time threshold tau=0.02 gives better raw perplexity but collapses entropy below 6.5, indicating diversity loss.
Limitations
- Energy compass is trained once and frozen; its effectiveness may decline as the student's flow-state distribution shifts.
- Activation threshold tau is fixed; adaptive schedules unexplored.
- Training overhead of 2.2x relative to FS-DFM at default settings.
Experimental setup
Datasets: FineWeb-Edu WikiText-103 GSM8K
Benchmarks: WikiText-103 GSM8K
Metrics: perplexity generation entropy energy ranking accuracy monotonicity
Baselines: FS-DFM DFM teacher (1024 steps) Duo SDTT ReMDM MDLM SEDD
Builds on: FS-DFM discrete flow matching SDTT Duo
Closest papers by meaning
- Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective (ICLR 2025 · similarity 0.88)
- Beyond Trajectory Matching: Reflow with Marginal Distribution Alignment (NeurIPS 2026 · similarity 0.87)
- Safe Few-Step Generation via Velocity Editing (NeurIPS 2026 · similarity 0.87)
- Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech (NeurIPS 2026 · similarity 0.87)
- Path-Guided Flow Matching for Dataset Distillation (NeurIPS 2026 · similarity 0.87)
- Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.