Research map / Architectures and efficiency / Attention and transformers
Layer-Wise Representation Fusion for Compositional Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LRF, which adds a fuse-attention module to each encoder and decoder layer of the Transformer so layers can attend to all previous layers' outputs, improving compositional generalization on CoGnition (MT) and CFQ (semantic parsing).
Problem
Neural seq2seq models fail at compositional generalization because syntactic and semantic representations in the top encoder and decoder layers are entangled, which the authors attribute to shallow residual connections that forget earlier layers' information.
Approach
Insert a fuse-attention module in every encoder and decoder layer that attends over all previous layers' outputs (localness enforced by masks), letting each layer adaptively fuse information from earlier layers instead of relying on shallow residual paths or fixed accumulation.
Key results
- On CoGnition, LRF reduces instance-level CTER from 28.4% (Transformer) to 20.0% and aggregate-level CTER from 62.9% to 50.3%, with BLEU 61.8.
- On CFQ, LRF raises RoBERTa's MCD-mean exact match from 43.4 to 52.4, but HPD (67.3) and Dangle (66.1) remain higher on average.
- Fuse-attention only at the top layer (LRF-onlytop) gives aggregate CTER 52.7%, worse than full LRF at 50.3%; naive accumulation (Transformer-accu) worsens to 71.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layer-wise fuse-attention to the Transformer | CoGnition CG-test · CoGnition CG-test | Instance-level CTER: 28.4% -> 20.0% | Helped |
| Add layer-wise fuse-attention to the Transformer | CoGnition CG-test · CoGnition CG-test | Aggregate-level CTER: 62.9% -> 50.3% | Helped |
| Add layer-wise fuse-attention to RoBERTa | CFQ · CFQ MCD-Mean | Exact-match accuracy: 43.4 -> 52.4 | Helped |
| Add fuse-attention on the encoder side only (ablation) | Transformer · CoGnition CG-test | Instance-level CTER: -7.2% | Helped |
| Add fuse-attention on the decoder side only (ablation) | Transformer · CoGnition CG-test | Instance-level CTER: -2.6% | Helped |
| Add fuse-attention on both encoder and decoder sides (ablation) | Transformer · CoGnition CG-test | Aggregate-level CTER: -12.6% | Helped |
| Accumulate previous-layer representations instead of adaptively fusing them (ablation) | Transformer · CoGnition CG-test | Instance-level CTER: 28.4% -> 37.3% | Hurt |
| Apply fuse-attention only in the top encoder and decoder layers instead of each layer (ablation) | LRF · CoGnition CG-test | Instance-level CTER: 20.0% -> 22.1% | Hurt |
| Increase Transformer depth to 8 encoder and 8 decoder layers (ablation) | Transformer · CoGnition CG-test | Aggregate-level CTER: -2.7% | Helped |
| Replace absolute with relative positional embeddings (ablation) | Transformer · CoGnition CG-test | Aggregate-level CTER: +1.6% | Hurt |
| Use dynamic linear combination of layers | Transformer · CoGnition CG-test | Aggregate-level CTER: +5.0% | Hurt |
Surprising findings
- Naively accumulating previous layers' outputs (Transformer-accu) degrades compositional generalization below the baseline Transformer.
- On CFQ, LRF underperforms HPD and Dangle on average, and its gains are more moderate than on CoGnition.
Limitations
- On CFQ, LRF is below HPD and Dangle on MCD-mean, and gains are smaller than on MT.
- Adds about 12M parameters, so some gains may partly reflect added capacity, though Transformer-Deep only slightly improves.
Experimental setup
Datasets: CoGnition CFQ
Benchmarks: CoGnition CFQ
Metrics: compound translation error rate (instance-level) compound translation error rate (aggregate-level) BLEU exact match accuracy
Baselines: Transformer Transformer-Rela Transformer-Small Transformer-Deep Bow SeqMix Dangle Proto-Transformer Transformer+CReg DLCL LSTM+attention Universal Transformer
Builds on: Transformer DLCL RoBERTa Fairseq
Closest papers by meaning
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.89)
- In-Context Compositional Learning vis Sparse Coding Transformer (NeurIPS 2025 · similarity 0.87)
- Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products (NeurIPS 2024 · similarity 0.87)
- Compositional Generalization via Forced Rendering of Disentangled Latents (ICML 2025 · similarity 0.87)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.86)
- Does Data Scaling Lead to Visual Compositional Generalization? (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.