Research map / Large language models / Language, safety and interpretability

SLMRec: Distilling Large Language Models into Small for Sequential Recommendation

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows intermediate LLM layers are redundant for sequential recommendation and distills a deep LLM-based E4SRec teacher into a 4-layer student (SLMRec) with feature-level losses, matching a 32-layer model at ~13% parameters and up to 6.6x/8.0x speedups.

Problem

LLM-based sequential recommenders are too large (7B+ parameters) to deploy efficiently on industrial platforms, and it is unclear how much LLM depth is actually needed for sequential recommendation.

Approach

Find that intermediate LLaMA layers are largely redundant for sequential recommendation, then distill a deeper E4SRec teacher into a shallow LLM student using cosine and L2 hidden-state feature losses plus multi-layer supervision adapters, with a theoretical argument that stacked attention layers can be approximated by a single layer.

Key results

• SLMRec (4 student layers from 8-layer teacher) reports the best average performance on Cloth and Movie, e.g., HR@1 16.69 vs E4SRec 16.71 on Cloth and 15.29 vs 14.74 on Movie for HR@1 (Movie).
• SLMRec uses about 13% of E4SRec's parameters and achieves 6.6x training and 8.0x inference speedups over E4SRec.
• The 4-layer student improves over its 8-layer E4SRec teacher by roughly 8% in reported performance.

Measured effects reported in the paper

Surprising findings

• Most intermediate LLM layers are redundant for sequential recommendation; an 8-layer E4SRec performs nearly as well as a 24-layer one.
• Directly using intermediate-layer representations without training gives poor recommendation performance.

Limitations

• Model cannot adapt to new scenarios via few-shot learning and must be retrained on the full dataset when new data arrives.
• Gains depend on a pretrained teacher and an ID embedding layer pretrained by SASRec; the theoretical justification is a simplified single-layer linearized analysis.

Experimental setup

Datasets: Amazon18 Cloth Amazon18 Movie Amazon18 Music Amazon18 Sport

Benchmarks: Amazon18 Cloth Amazon18 Movie Amazon18 Music Amazon18 Sport

Metrics: HR@1 HR@5 NDCG@5 MRR training time inference time

Baselines: GRU4Rec Caser HGN BERT4Rec SASRec LightSANs S^3-Rec DuoRec MAERec Open-P5 E4SRec

Builds on: E4SRec SASRec LoRA Open-P5

Closest papers by meaning

• LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.93)
• LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.92)
• Customizing Language Models with Instance-wise LoRA for Sequential Recommendation (NeurIPS 2024 · similarity 0.91)
• Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (NeurIPS 2025 · similarity 0.90)
• LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation (AAAI 2025 · similarity 0.90)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.89)