Research map / Architectures and efficiency / Attention and transformers
Characterizing intrinsic compositionality in transformers with Tree Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCI-minimizing tree projections instead of a random tree | Tree projections · COGS IID validation | bracketing F1: 75.6 vs 44.9 | Helped |
| Use SCI-minimizing tree projections instead of a random tree | Tree projections · M-PCFGSET IID validation | bracketing F1: 46.5 vs 26.8 | Helped |
| Use SCI-minimizing tree projections instead of a random tree | Tree projections · GeoQuery IID validation | bracketing F1: 48.0 vs 41.0 | Helped |
| Use SCI-minimizing tree projections instead of left-branching trees | Tree projections · COGS IID validation | bracketing F1: 75.6 vs 30.2 | Helped |
| Use SCI-minimizing tree projections instead of left-branching trees | Tree projections · M-PCFGSET IID validation | bracketing F1: 46.5 vs 17.9 | Helped |
| Use SCI-minimizing tree projections instead of left-branching trees | Tree projections · GeoQuery IID validation | bracketing F1: 48.0 vs 26.0 | Helped |
| Use SCI-minimizing tree projections instead of right-branching trees | Tree projections · COGS IID validation | bracketing F1: 75.6 vs 75.2 | Helped |
| Use SCI-minimizing tree projections instead of right-branching trees | Tree projections · M-PCFGSET IID validation | bracketing F1: 46.5 vs 58.1 | Hurt |
| Use SCI-minimizing tree projections instead of right-branching trees | Tree projections · GeoQuery IID validation | bracketing F1: 48.0 vs 69.7 | Hurt |
| Train with sequence-transduction supervision rather than masked language modeling (ablation) | Transformer encoders · COGS and GeoQuery | tree-projection alignment with gold syntax | Helped |
| Train with sequence-transduction supervision rather than masked language modeling (ablation) | Transformer encoders · COGS and GeoQuery | tree-structuredness | Mixed |
| Train the encoder on the task over training rather than using an earlier checkpoint | Transformer encoders · COGS and M-PCFGSET | tree-projection alignment with gold syntax | Helped |
Closest papers by meaning
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.90)
- Towards Equipping Transformer with the Ability of Systematic Compositionality (AAAI 2024 · similarity 0.88)
- How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.88)
- Compositional Reasoning with Transformers, RNNs, and Chain of Thought (NeurIPS 2025 · similarity 0.87)
- Banyan: Improved Representation Learning with Explicit Structure (ICML 2025 · similarity 0.87)
- Emergence of a High-Dimensional Abstraction Phase in Language Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.