Research map / Large language models / Instruction and fine-tuning
Customizing Language Models with Instance-wise LoRA for Sequential Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iLoRA, which splits LoRA matrices into MoE experts and uses a sequence-representation-guided gate (from SASRec) to build instance-specific LoRA weights for LLM-based sequential recommendation.
Problem
Uniform LoRA fine-tuning of LLMs for sequential recommendation fails to capture individual user behavior variability, causing negative transfer between disparate user sequences.
Approach
Treat each user sequence as a task and integrate MoE into LoRA: the up- and down-projection matrices are split into K experts, and a gating network conditioned on a SASRec sequence embedding produces softmax weights that compose an instance-wise LoRA update, keeping the parameter count equal to standard LoRA.
Key results
- iLoRA achieves HitRatio@1 of 0.5000 on LastFM, 0.5275 on MovieLens, and 0.5264 on Steam, the best among compared methods including LLaRA (0.4508, 0.4421, 0.4949).
- The paper reports an average 11.4% relative HitRatio@1 improvement over basic LoRA with less than 1% relative increase in trainable parameters.
- Using 4 experts gives the best performance across the three datasets; 8 experts yields similar or slightly lower results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform LoRA with instance-wise LoRA | Llama2-7B sequential recommendation · LastFM | HitRatio@1: 0.4508 -> 0.5000 | Helped |
| Replace uniform LoRA with instance-wise LoRA | Llama2-7B sequential recommendation · MovieLens | HitRatio@1: 0.4421 -> 0.5275 | Helped |
| Replace uniform LoRA with instance-wise LoRA | Llama2-7B sequential recommendation · Steam | HitRatio@1: 0.4949 -> 0.5264 | Helped |
| Use sequence representations rather than random-initialized or token-collapsed embeddings to guide the gate (ablation) | iLoRA · LastFM, MovieLens, and Steam | HitRatio@1 | Helped |
| Use 4 experts rather than 2 experts (ablation) | iLoRA · LastFM | HitRatio@1: 5.2% | Helped |
| Use 4 experts rather than 2 experts (ablation) | iLoRA · MovieLens | HitRatio@1: 6.4% | Helped |
| Use 4 experts rather than 2 experts (ablation) | iLoRA · Steam | HitRatio@1: 2.2% | Helped |
| Increase the number of experts from 4 to 8 (ablation) | iLoRA · LastFM, MovieLens, and Steam | HitRatio@1 | Mixed |
| Use 2 experts rather than the default 4 experts (ablation) | iLoRA · Steam and MovieLens | HitRatio@1 | No clear effect |
| Use 2 experts rather than the default 4 experts (ablation) | iLoRA · LastFM | HitRatio@1 | Hurt |
| Increase the number of experts to 8 (ablation) | iLoRA on Steam · Steam | HitRatio@1 | No clear effect |
Surprising findings
- Gradient similarity analysis of LLaRA shows misalignment between sequences distant in collaborative space, with some pairs at -0.75 cosine similarity.
- Increasing experts to 8 does not improve and may cause premature overfitting, per the authors' speculation.
Limitations
- Limited exploration of larger expert combinations due to computational resources.
- Hard routing with many experts not extensively investigated.
- Applicability beyond sequential recommendation not explored.
Experimental setup
Datasets: LastFM MovieLens Steam
Benchmarks: LastFM MovieLens Steam
Metrics: HitRatio@1 ValidRatio
Baselines: GRU4Rec Caser SASRec Llama2-7B GPT-4 ChatRec MoRec TALLRec LLaRA
Builds on: LoRA LLaRA TALLRec SASRec Mixture of Experts
Closest papers by meaning
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.91)
- LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.89)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.89)
- Augmenting Sequential Recommendation with Balanced Relevance and Diversity (AAAI 2025 · similarity 0.88)
- CoRA: Collaborative Information Perception by Large Language Model’s Weights for Recommendation (AAAI 2025 · similarity 0.88)
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.