Research map / Large language models / Language, safety and interpretability

Improving LLMs for Recommendation with Out-Of-Vocabulary Tokens

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes META ID, which builds user and item identifiers from clustered meta-path skip-gram embeddings as out-of-vocabulary tokens added to an LLM's vocabulary, improving LLM-based recommendation with T5 and LLaMA2-7b.

Problem

LLM-based recommenders represent users and items with in-vocabulary numeric tokens, which lack expressive power and cause mapping conflicts, weakening recommendation performance after fine-tuning.

Approach

Learn user and item embeddings from meta-path random walks (U-I-U with equal ratings) via a skip-gram model, cluster them with K-Means into hierarchical OOV token groups, add these tokens to the LLM vocabulary, and initialize them with a linear layer from cluster centroids. The paper also introduces memorization and diversity scores to evaluate ID representations.

Key results

• On sequential recommendation, META ID (T5) reaches H@5 0.0510 and N@10 0.0429 on Beauty, versus 0.0404 H@5 for the best P5-based baseline SID on Beauty, and outperforms SASRec and other baselines on Sports and Beauty.
• On direct recommendation, META ID achieves the best results on all three datasets, e.g. H@10 0.0803 on Toys versus 0.0340 for CID on Beauty and 0.0342 for CID on Sports.
• Ablation: K-Means grouping outperforms DBSCAN, Spectral and RQ-VAE on most metrics, and combined user-and-item indexing outperforms user-only or item-only indexing.

Measured effects reported in the paper

Surprising findings

• Randomly re-initializing numeric tokens does not help and is detrimental on Sports and Toys, suggesting pretrained numeric token embeddings cannot simply be discarded.
• Independent per-item OOV tokens (IID) perform far worse than META ID, despite having one token per item, due to slow training and poor learning.

Limitations

• Uses a fixed database of users and items, so newly appearing items are not handled.
• Experiments are limited to small Amazon datasets and two backbones (T5-small and LLaMA2-7b).

Experimental setup

Datasets: Amazon Sports&Outdoors Amazon Beauty Amazon Toys&Games

Benchmarks: Amazon Sports Amazon Beauty Amazon Toys

Metrics: HR@K NDCG@K RMSE BLEU-1 BLEU-4 memorization score diversity score

Baselines: Caser HGN GRU4Rec BERT4Rec FDSA SASRec S3-Rec CL4SRec P5 with RID P5 with SID P5 with CID

Builds on: P5 RQ-VAE LoRA skip-gram

Closest papers by meaning

• Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (NeurIPS 2025 · similarity 0.91)
• LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.90)
• Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenization (AAAI 2025 · similarity 0.90)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.90)
• Language Representations Can be What Recommenders Need: Findings and Potentials (ICLR 2025 · similarity 0.89)
• LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation (AAAI 2025 · similarity 0.89)