Research map / Large language models / Language, safety and interpretability
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge
Xutao Ma, Yixiao Huang, Hanlin Zhu, Somayeh Sojoudi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add identity bridge data (A->A) to forward-only training data | Llama-3.2-1B-Instruct finetuning · Husband-Wife and Parent-Child reversal | reversal test accuracy: near-zero -> 50% | Improved |
| add identity bridge data to one-layer transformer training | one-layer transformer · symbolic reversal task N=10 | reversal test MRR | Improved |
| phrase identity bridge directly (IDN) instead of rewriting to OCR form (ablation) | Identity Bridge on Llama-3.2-1B · Husband-Wife reversal | reversal test accuracy | Hurt |
| repeat forward relation data k=6 times in OCR format (ablation) | Identity Bridge OCR format · Husband-Wife reversal | reversal test accuracy: 50% (OCR-6, highest) | Improved |
| use shorter entity names (one-token numbers versus three-token names) (ablation) | Identity Bridge OCR format · Husband-Wife reversal | reversal test accuracy: 10% -> nearly 100% | Improved |
| use moderate weight decay (0.20 to 0.40) rather than 0.15 or 0.45 (ablation) | Identity Bridge OCR format · Husband-Wife reversal | reversal test accuracy: 0.12 (wd 0.15), 0.55 (wd 0.30), 0.33 (wd 0.45) | Mixed |
| train on forward relation only with factorized one-layer transformer (ablation) | one-layer transformer (theory) · symbolic reversal task | reversal margin: margin = 0 | Hurt |
| model learns a shortcut copying the name token | Identity Bridge on Llama-3.2-1B · Husband-Wife shortcut test | shortcut test accuracy: near 100% early, around 60% for IDN | Hurt |
Closest papers by meaning
- Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics (NeurIPS 2024 · similarity 0.92)
- A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse (NeurIPS 2026 · similarity 0.91)
- The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (ICLR 2024 · similarity 0.88)
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.88)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.88)
- The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
