Research map / Label-efficient and robust learning / Self-supervised and contrastive
Language Models as Semantic Indexers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMIndexer, a self-supervised T5-based semantic indexer that learns hierarchical discrete semantic IDs via document reconstruction, progressive training, and contrastive loss, and evaluates them on recommendation, product search, and document retrieval.
Problem
Existing semantic ID learning uses a two-stage pipeline (off-the-shelf text embeddings then rq-VAE or hierarchical clustering), which loses information and mismatches embedding distributions; learning semantic IDs end-to-end is hard due to discreteness, sequential structure, and lack of semantic sup
Approach
Learn semantic IDs directly from documents with a generative encoder-decoder LM: codebook lookups at each ID position, trained with a self-supervised reconstruction objective (a shallow reconstructor predicts masked tokens from the IDs plus hint tokens), progressive per-position training with reducing hints, a contrastive loss to separate documents sharing prefixes, and a commitment loss, plus war
Key results
- On Amazon ID quality (AMI), LMIndexer reaches 0.3563 / 0.4163 / 0.3536 on Beauty / Sports / Toys, above the best rq-VAE indexer baselines (0.3100 / 0.2774 / 0.3154).
- On next-item recommendation, LMIndexer achieves Recall@5 of 0.0415 on Amazon-Beauty versus 0.0325 for the best traditional baseline HGN.
- On NQ320k document retrieval, LMIndexer gets Recall@1 0.6631 vs 0.6480 for rq-VAE indexer, but DPR remains best on MS MACRO 1M MRR@10 (0.5561 vs 0.5485 for LMIndexer).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LMIndexer instead of the strongest reported baseline | Sequential recommendation · Amazon-Beauty, Amazon-Sports, and Amazon-Toys | Recall@5 and NDCG@5: 0.0325 -> 0.0415; 0.0189 -> 0.0222; 0.0321 -> 0.0404 (Recall@5); 0.0206 -> 0.0262; 0.0122 -> 0.0142; 0.0221 -> 0.0268 (N | Helped |
| Use LMIndexer instead of the strongest reported baseline | Product search · Amazon-Beauty, Amazon-Sports, and Amazon-Toys | NDCG@5 and MAP@5: 0.2710 -> 0.3187; 0.2606 -> 0.2870; 0.2805 -> 0.2865 (NDCG@5); 0.2469 -> 0.2888; 0.2354 -> 0.2607; 0.2420 -> 0.2592 (MAP | Helped |
| Use LMIndexer instead of the strongest reported baseline | Document retrieval · NQ320k, TREC-DL 1M, and MACRO 1M | Recall@1, Recall@10, NDCG@10, and MRR@10: 0.6480 -> 0.6631; 0.8322 -> 0.8589; 0.4265 -> 0.4519; 0.4579 -> 0.4695; 0.5561 -> 0.5485 | Mixed |
| Remove commitment loss (ablation) | LMIndexer · Amazon-Sports | perplexity of previously learned IDs: 305.39 -> 147.10 | Hurt |
| Remove commitment loss (ablation) | LMIndexer · Amazon-Toys | perplexity of previously learned IDs: 280.30 -> 211.60 | Hurt |
| Remove commitment loss (ablation) | LMIndexer · Amazon-Beauty | perplexity of previously learned IDs: 287.01 -> 261.04 | Hurt |
| Increase reconstructor depth from 1 to 2 layers (ablation) | LMIndexer · Amazon-Beauty | ID quality (AMI): 0.3563 -> 0.2390 | Hurt |
| Increase reconstructor depth from 1 to 2 layers (ablation) | LMIndexer · Amazon-Beauty | Recommendation Recall@5: 0.0415 -> 0.0284 | Hurt |
| Increase reconstructor depth from 1 to 2 layers (ablation) | LMIndexer · Amazon-Beauty | Retrieval NDCG@5: 0.3187 -> 0.2528 | Hurt |
| Increase semantic ID length (ablation) | LMIndexer · Amazon-Beauty recommendation | recommendation performance | Helped |
| Increase codebook size (ablation) | LMIndexer · Amazon-Beauty recommendation | downstream task performance | Helped |
| Add contrastive loss (ablation) | LMIndexer · Amazon-Sports semantic ID training | difference ratio for documents sharing an ID prefix | Helped |
Surprising findings
- A dual encoder (DPR) outperforms semantic indexer baselines in many product search and document retrieval cases, and remains best on the large MS MACRO 1M corpus.
- Making the reconstructor more powerful (2 or 3 layers) degrades semantic ID quality and downstream performance, because knowledge shifts into the reconstructor.
Limitations
- LMIndexer does not beat the dual encoder DPR on the large MS MACRO 1M corpus.
- Experiments use only the small T5-base backbone and short ID length T=3 with limited codebook sizes.
- Some training stages (reconstructor warm-up, codebook initialization) add procedural complexity and sensitivity to hyperparameters such as hint ratio.
Experimental setup
Datasets: Amazon-Beauty Amazon-Sports Amazon-Toys Amazon product search dataset NQ320k MS MACRO 1M TREC-DL 1M
Benchmarks: Amazon-Beauty Amazon-Sports Amazon-Toys NQ320k TREC-DL 1M MS MACRO 1M
Metrics: AMI Recall@5 NDCG@5 NDCG@5 (search) MAP@5 Recall@1 Recall@10 NDCG@10 MRR@10 Macro-F1
Baselines: rq-VAE indexer hierarchical clustering indexer HGN GRU4Rec BERT4Rec FDSA bm25 Dual Encoder (DPR) SEAL
Builds on: T5 rq-VAE DSI SEAL
Closest papers by meaning
- Rethinking Semantic ID Construction for Generative Recommendation: SimHash with Parallel Decoding and Semantic Alignment (NeurIPS 2026 · similarity 0.87)
- Descriptive and Discriminative Document Identifiers for Generative Retrieval (AAAI 2025 · similarity 0.86)
- DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (AAAI 2025 · similarity 0.85)
- Generative Retrieval Meets Multi-Graded Relevance (NeurIPS 2024 · similarity 0.85)
- Learning to Rank in Generative Retrieval (AAAI 2024 · similarity 0.83)
- Select and Augment: Enhanced Dense Retrieval Knowledge Graph Augmentation (Abstract Reprint) (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.