Research map / Large language models / Retrieval and knowledge
Active Large Language Model-Based Knowledge Distillation for Session-Based Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALKDRec, an active-learning strategy that selects a small subset of session instances for GPT-4-turbo teacher predictions, then distills them into lightweight session-based recommenders (FPMC, STAMP, AttMix).
Problem
LLM-based knowledge distillation for session-based recommendation is costly because LLMs must predict for all sessions, and LLM predictions can be ineffective (similar or incorrect) on many instances.
Approach
Select a small set of instances via a max-min expected-gain active learning policy that weights instances by difficulty (negative consistency with the conventional teacher) and by their predicted effect type (effective, similar, incorrect), so that only informative instances are sent to the LLM teacher.
Key results
- ALKDRec improves over the strongest KD baseline by average 17.03% (FPMC), 9.81% (STAMP), and 12.97% (AttMix) across the reported metrics.
- On Hetrec2011-ML with FPMC, ALKDRec reaches recall@10 0.02150 and ndcg@10 0.01019, versus 0.01933-level baselines such as DLLM2Rec's 0.01623 and FTD's 0.01866 recall@10 for the best baseline.
- ALKDRec with a 10-dim student outperforms a 100-dim teacher recommender in the paper's comparisons, and its training uses about 44 minutes and 8.6 USD versus 1782 minutes and 347.0 USD for all samples on Amazon-Games.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALKDRec's LLM-guided active instance selection rather than the strongest KD baseline | FPMC, STAMP, and AttMix · Hetrec2011-ML and Amazon-Games | recall@10: 0.01866 -> 0.02150; 0.01501 -> 0.01623; 0.01014 -> 0.01095; 0.05200 -> 0.05748; 0.05051 -> 0.05325; 0.02215 -> 0.02936 | Helped |
| Add LLM-teacher distillation rather than distilling from the conventional teacher recommender (ablation) | ALKDRec with FPMC · Hetrec2011-ML | recall@10: 0.01907 -> 0.02150 | Helped |
| Add LLM-teacher distillation rather than distilling from the conventional teacher recommender (ablation) | ALKDRec with FPMC · Amazon-Games | recall@10: 0.05175 -> 0.05748 | Helped |
| Select instances with ALKDRec's active policy rather than random sampling (ablation) | ALKDRec with FPMC · Hetrec2011-ML | recall@10: 0.01907 -> 0.02150 | Helped |
| Select instances with ALKDRec's active policy rather than random sampling (ablation) | ALKDRec with FPMC · Amazon-Games | recall@10: 0.05101 -> 0.05748 | Helped |
| Select the easiest instances rather than using ALKDRec's active policy (ablation) | ALKDRec with FPMC · Hetrec2011-ML | recall@10: 0.02150 -> 0.01623 | Hurt |
| Select the easiest instances rather than using ALKDRec's active policy (ablation) | ALKDRec with FPMC · Amazon-Games | recall@10: 0.05748 -> 0.05076 | Hurt |
| Select the hardest instances rather than using ALKDRec's active policy (ablation) | ALKDRec with FPMC · Hetrec2011-ML | recall@10: 0.02150 -> 0.01298 | Hurt |
| Select the hardest instances rather than using ALKDRec's active policy (ablation) | ALKDRec with FPMC · Amazon-Games | recall@10: 0.05748 -> 0.04877 | Hurt |
| Replace ALKDRec's ranking-task selection with RAD-BC (ablation) | ALKDRec with FPMC · Hetrec2011-ML | recall@10: 0.02150 -> 0.01420 | Hurt |
| Replace ALKDRec's ranking-task selection with RAD-BC (ablation) | ALKDRec with FPMC · Amazon-Games | recall@10: 0.05748 -> 0.05200 | Hurt |
Surprising findings
- Hardest-instance and easiest-instance selection both perform poorly, and some KD baselines (e.g., STAMP and AttMix on Amazon-Games) underperform the vanilla student.
- The ALKDRec gain on AttMix recall@5 in Hetrec2011-ML is negative (-6.25%) relative to the best baseline.
Limitations
- Gain values are hand-designed heuristics (exponential rank-based gain, g_si = g_in = g_ef/2) and the 1:5:4 effective/similar/incorrect ratio is set by grid search.
- Evaluated only on two datasets and three relatively simple backbones (FPMC, STAMP, AttMix).
Experimental setup
Datasets: Hetrec2011-ML Amazon-Games
Benchmarks: Hetrec2011-ML Amazon-Games
Metrics: recall@5 recall@10 ndcg@5 ndcg@10
Baselines: DE FTD HTD unKD DSL DLLM2Rec TR Random RAD-BC
Builds on: DLLM2Rec RAD-BC robust active distillation topology distillation
Closest papers by meaning
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.89)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.87)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.87)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.87)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.87)
- LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.