Research map / Architectures and efficiency / Attention and transformers
Towards Equipping Transformer with the Ability of Systematic Compositionality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT, a BERT-style encoder with a multi-primitive composition module (sparse-attention over a learned sememe-like codebook) and two pre-training tasks (sememe-guided decomposition and semantics composition), evaluated on phrase/sentence compositionality, OOD generalization, noise robustness, and GLUE.
Problem
Transformers such as BERT struggle to generalize to composed, unseen combinations of seen primitives (systematic compositionality), e.g., distinguishing phrases built from the same words in different orders.
Approach
Equip a Transformer encoder with a learnable discrete codebook of primitives and a differentiable sparse-attention decomposition that composes multiple codes into a compositional representation, mixed with the contextual representation, and supervise the codebook with OpenHowNet sememes plus new pre-training losses.
Key results
- CAT_mix improves on BERT by +1.10 on PAWS (swap-only) and +6.42 average on phrase correlation (BiRD/BiRD-ABBA) per the paper's averages.
- CAT_mix outperforms BERT by 1.83 on average on out-of-distribution datasets and CAT_mix outperforms BERT by 3.09 on SQuAD-adversarial.
- On GLUE, CAT_cont averages 76.60 vs BERT 76.07, and CAT_mix averages 76.16, with minimal impact on standardized tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAT_mix instead of BERT | CAT with mixed representation · PAWS (SO) | accuracy: 88.22 -> 89.32 | Helped |
| Use CAT_comp instead of BERT | CAT with compositional representation · BiRD | Pearson correlation: 19.50 -> 34.44 | Helped |
| Use CAT_mix instead of BERT | CAT with mixed representation · BiRD-ABBA | Pearson correlation: 1.64 -> 6.25 | Helped |
| Use mixed representation instead of contextual representation (ablation) | CAT · BiRD | Pearson correlation: 20.19 -> 27.72 | Helped |
| Use compositional representation instead of mixed representation (ablation) | CAT · BiRD | Pearson correlation: 27.72 -> 34.44 | Helped |
| Use CAT_mix instead of BERT | CAT with mixed representation · SST2 OOD | accuracy: 76.61 -> 79.36 | Helped |
| Use CAT_mix instead of BERT | CAT with mixed representation · MNLI-Letters OOD | accuracy: 72.79 -> 73.96 | Helped |
| Use CAT_mix instead of BERT | CAT with mixed representation · AMAZON-Video OOD | accuracy: 65.16 -> 67.44 | Helped |
| Use compositional representation instead of contextual representation (ablation) | CAT · GLUE average | average score: 76.60 -> 73.48 | Hurt |
| Use mixed representation instead of contextual representation (ablation) | CAT · GLUE average | average score: 76.07 -> 76.16 | No clear effect |
| Rely on compositional representation rather than mixed representation (ablation) | CAT · SQuAD-adversarial | accuracy | Helped |
| Vary fine-tuning sample size using compositional representations (ablation) | CAT_comp · IMDB and SST2 | accuracy | Mixed |
Surprising findings
- ChatGPT scores only 23.28 Pearson on BiRD-ABBA versus 45.65 on BiRD, indicating poor alignment with human phrase similarity on word-order-swapped pairs.
- Sample-efficiency experiments found no consistent advantage for discrete compositional representations over contextual ones.
Limitations
- Only a 4-layer BERT-sized model was pre-trained due to compute limits; scaling is untested.
- Evaluation is restricted to discriminative, BERT-style tasks and does not test generative LLMs.
- Compositional representation alone underperforms on standardized GLUE tasks, and the approach relies on OpenHowNet sememe annotations (Chinese-centric resource).
Experimental setup
Datasets: BooksCorpus English Wikipedia OpenHowNet PAWS-QQP PAWS (swap-only) BiRD BiRD-ABBA IMDB SST2 STSB MNLI AMAZON reviews
Benchmarks: PAWS (swap-only) BiRD BiRD-ABBA IMDB/SST2 OOD STSB image/MSRvid OOD MNLI telephone/letters OOD AMAZON music/video OOD SQuAD-adversarial GLUE
Metrics: accuracy Pearson correlation F1 Spearman correlation
Baselines: BERT RoBERTa DistilBERT
Builds on: BERT RoBERTa DistilBERT vector quantization rectified linear attention
Closest papers by meaning
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.87)
- In-Context Compositional Learning vis Sparse Coding Transformer (NeurIPS 2025 · similarity 0.85)
- Limits of Transformer Language Models on Learning to Compose Algorithms (NeurIPS 2024 · similarity 0.85)
- Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures (ICLR 2025 · similarity 0.85)
- Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.85)
- Assign and Add: A Mechanistic Study of Compositional Arithmetic (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.