Research map / Architectures and efficiency / Attention and transformers
Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure
Boshi Wang, Huan Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Represent inputs at the abstract concept level (own embedding per entity) instead of surface names (ablation) | Standard decoder-only transformer · Reversal task | MRR: 0.0% accuracy (surface) -> MRR 0.951 (concept level, |E_A|=50K) | Improved |
| Increase |E_A| (number of training entities) (ablation) | Standard transformer, concept level · Abstract reversal | MRR: 0.823 -> 0.964 (1 layer, 2.5K -> 100K) | Improved |
| Use JEPA with in-batch contrastive loss at concept level | Autoregressive transformer · Surface-level reversal | MRR: 0.0% -> 0.864 | Improved |
| Increase depth of recognition module or semantic-to-recognition ratio in JEPA (ablation) | JEPA · Surface-level reversal | MRR: 0.864 -> 0.230 | Hurt |
| Increase name multiplicity (more entities sharing tokens) (ablation) | JEPA · Surface-level reversal | MRR | Hurt |
| Increase model width from 768 to 1280 (ablation) | JEPA · Surface-level reversal | MRR | Improved |
| Replace last MLP of recognition module with a memory layer (ablation) | JEPA · Surface-level reversal | MRR | Improved |
| Add memory layers to a standard transformer without JEPA (ablation) | Standard transformer · Surface-level reversal | Accuracy | No clear effect |
| Vary depth or tied vs untied embeddings in standard surface-level training (ablation) | Standard transformer · Surface-level reversal | Accuracy (%): 0.0% | No clear effect |
| Use parametric memory with JEPA and memory layers vs frontier LLMs with in-context facts | Large-scale arithmetic reasoning · Branching factor 10 to 40 | Accuracy | Improved |
Closest papers by meaning
- Delving into the Reversal Curse: How Far Can Large Language Models Generalize? (NeurIPS 2024 · similarity 0.87)
- Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers (NeurIPS 2024 · similarity 0.87)
- Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics (NeurIPS 2024 · similarity 0.87)
- Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers (NeurIPS 2025 · similarity 0.87)
- You Can’t Have It Both Ways: Concept Entanglement Limits Diffusion Model Unlearning (NeurIPS 2026 · similarity 0.86)
- Interaction Asymmetry: A General Principle for Learning Composable Abstractions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
