Research map / Large language models / Reasoning and chain of thought

Efficient Inference for Large Language Model-based Generative Recommendation

ICLR 2025 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AtSpeed, an alignment framework for speculative decoding in LLM-based generative recommendation under N-to-K (top-K beam) verification, with a strict-verification objective (AtSpeed-S) and a relaxed sampling verification with TVD alignment (AtSpeed-R), achieving ~2x walltime speedup on Amazon Beauty and Games.

Problem

Speculative decoding for LLM-based generative recommendation is hard because beam search must return top-K distinct item sequences, so every step requires the draft model to cover all top-K sequences of the target LLM (N-to-K verification), limiting acceptance and speedup.

Approach

Align the draft model to the target LLM's top-K sequence distribution: under strict top-K verification, minimize a reverse-KL alignment loss with a density regularizer over constrained top-K tokens (AtSpeed-S); under a relaxed sampling verification that accepts high-probability non-top-K drafts with probability p/q, minimize TVD on top-K sequences (AtSpeed-R), with tree-based attention for efficie

Key results

• AtSpeed-S under strict top-K verification achieves average walltime speedup of 1.97x on Beauty and 1.83x on Games with identical top-K results.
• AtSpeed-R under relaxed sampling verification reaches average speedup of 2.11x on Beauty and 2.05x on Games.
• Relaxed sampling verification generally yields higher speedup than strict top-K verification across methods, with only limited Recall@5/NDCG@5 drops.

Measured effects reported in the paper

Surprising findings

• Logit-level KD methods (WordKD, TVDKD) often perform comparable to or worse than plain SFT for the draft model.
• Using only the target LLM's top-1 sequence for AtSpeed-R lowers acceptance, since it lacks diversity.

Limitations

• Only evaluated on LC-Rec with LLaMA-7B target and LLaMA-68M draft; generalization to other generative recommenders is not shown.
• Relaxed verification trades a small accuracy drop for speed; the equivalence relies on a with-replacement sampling approximation.
• Draft model is an LLM-sized small model, so drafting overhead remains.

Experimental setup

Datasets: Amazon Beauty Amazon Games MovieLens-1M Goodreads

Benchmarks: Amazon Beauty Amazon Games MovieLens-1M Goodreads

Metrics: walltime speedup (WS@K) accept steps (AS@K) Recall@K NDCG@K

Baselines: DARE SFT WordKD TVDKD SeqKD

Builds on: speculative decoding DistillSpec LC-Rec SeqKD TVDKD

Closest papers by meaning

• Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.89)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.88)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.87)
• Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (ICML 2024 · similarity 0.87)
• Nearest Neighbor Speculative Decoding for LLM Generation and Attribution (NeurIPS 2024 · similarity 0.86)
• Accelerated Speculative Sampling Based on Tree Monte Carlo (ICML 2024 · similarity 0.86)