Research map / Large language models / Retrieval and knowledge
The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes the reversal curse as a "factorization curse" of autoregressive objectives and shows that factorization-agnostic training (uniform-rate masked LM, MLM-U) mitigates it across synthetic retrieval, BioS, WikiReversal, relationship, and star-graph planning tasks.
Problem
Language models trained with left-to-right next-token prediction fail to retrieve information in a different order than it was trained (the reversal curse), which contributes to hallucination and limits knowledge retrieval and planning.
Approach
Frames the reversal curse as a failure to learn the same joint distribution under different factorizations, and proposes factorization-agnostic objectives (uniform-rate masked language modeling, equivalent to a discrete absorbing-state diffusion / permutation LM objective) that train over all context/prediction splits, with an encoder-decoder architecture.
Key results
- On the synthetic retrieval task, AR and AR w/reverse reach 100% forward but 0% backward accuracy, while MLM-U reaches 100% on both directions.
- On BioS birth_date-to-full_name, MLM-U achieves 68% backward accuracy versus 0% for AR, AR w/reverse, and MLM.
- On WikiReversal with different train/test split seeds, a 100M MLM-U model reaches 66% forward and 46% backward exact match, versus 28%/6.2% for AR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MLM-U instead of AR | retrieval task · Retrieval task, backward | exact match accuracy: 0 -> 100 | Helped |
| train on token-reversed sequences | AR on retrieval task · Retrieval task, backward | exact match accuracy: 0 -> 0 | No clear effect |
| use MLM-U instead of fixed-rate MLM | retrieval task · Retrieval task, backward | exact match accuracy: 22 -> 100 | Helped |
| add delimiter tokens around entity-reversed sequences | AR w/reverse on relationship task · Relationship task | incorrect inference: 44 -> 0 | Helped |
| use MLM-U instead of AR | BioS · BioS, backward | exact match accuracy: 0 -> 68 | Helped |
| use MLM-U instead of AR | WikiReversal · WikiReversal, backward | exact match QA accuracy: 4.3 -> 7.9 | Helped |
| vary the masking rate instead of using a fixed rate (ablation) | MLM on retrieval task · Retrieval task, backward | per token accuracy: 0 -> 100 | Helped |
| use MLM-U instead of fixed-rate MLM (ablation) | BioS · BioS, backward | exact match accuracy: 0 -> 68 | Helped |
| increase fixed masking rate from 15% to 40% (ablation) | MLM on retrieval task · Retrieval task, backward | per token accuracy: 22 -> 16 | Hurt |
| increase fixed masking rate from 40% to 85% (ablation) | MLM on retrieval task · Retrieval task, backward | per token accuracy: 16 -> 28 | Helped |
| increase fixed masking rate from 15% to 85% (ablation) | MLM on BioS · BioS, backward | exact match accuracy: 0.00 -> 0.08 | Helped |
| increase fixed masking rate from 15% to 40% (ablation) | MLM on relationship task · Relationship task, forward | exact match accuracy: 24 -> 77 | Helped |
Surprising findings
- Reversing entity order without delimiters produces a high incorrect inference rate (44%) on the non-reciprocal relationship task.
- Backward accuracy under MLM-U keeps improving long after forward accuracy saturates (delayed generalization), e.g. beyond 300k steps on WikiReversal.
Limitations
- Factorization-agnostic objectives are harder to optimize and show delayed generalization, requiring long training.
- On the realistic WikiReversal benchmark MLM-U performance is not strong enough to claim success, possibly needing larger models.
- Experiments are largely on small models (about 100M parameters) trained from scratch, with only limited Mistral 7B LoRA comparisons.
Experimental setup
Datasets: BioS GenWiki WikiReversal synthetic key-value retrieval star graph task
Benchmarks: synthetic retrieval task relationship task BioS WikiReversal star graph task
Metrics: exact match accuracy forward accuracy backward accuracy incorrect inference rate
Baselines: autoregressive (AR) AR with reversed tokens MLM with fixed masking rate Mistral 7B (LoRA finetuned) teacher-forced AR on star graph
Builds on: XLNet permutation language modeling BERT masked language modeling discrete diffusion absorbing state models BioS synthetic biographies RHO-LOSS
Closest papers by meaning
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.89)
- Physics of Language Models: Part 3.2, Knowledge Manipulation (ICLR 2025 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.87)
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR 2024 · similarity 0.87)
- Retrieval Head Mechanistically Explains Long-Context Factuality (ICLR 2025 · similarity 0.87)
- SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.