Research map / Architectures and efficiency / Attention and transformers
In-Context Compositional Learning vis Sparse Coding Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates Transformer attention as sparse coding over learned encoding and decoding dictionaries, with soft-thresholded coefficients and linear coefficient transfer from context to target, evaluated on S-RAVEN, RAVEN, and small LLM fine-tuning experiments.
Problem
Transformers struggle with in-context compositional learning tasks because dense softmax attention gives no explicit structural inductive bias for representing and transferring compositional rules across examples.
Approach
Reinterpret attention as sparse coding: an input-dependent encoding dictionary (from W_qk) yields coefficients that are sparsified by soft-thresholding and recombine a decoding dictionary (from W_vo) to form outputs; target-task coefficients are estimated as a learned linear combination of context-task coefficients (lifting scheme).
Key results
- On S-RAVEN, the method outperforms standard Transformer and HYLA baselines, reaching 73.6% accuracy in the best basis-function variant versus 71.7% for the linear baseline in the ablation.
- On RAVEN (2x2, PSNR), the standard Transformer yields nearly 0% of test samples above PSNR 40 while the method retains around 40%.
- On commonsense reasoning with Llama-7B, adding the method to LoRA raises average accuracy from 75.6 to 76.3 and to DoRA from 77.8 to 78.1, using about 128 new parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sparse-coding attention to Llama-7B | Llama-7B · Commonsense reasoning benchmarks (average) | accuracy: 70.5 -> 71.9 | Helped |
| Add sparse-coding attention to DoRA | Llama-7B with DoRA · Commonsense reasoning benchmarks (average) | accuracy: 77.8 -> 78.1 | Helped |
| Add sparse-coding attention to LoRA | Llama-7B with LoRA · HellaSwag | accuracy: 83.4 -> 81.6 | Hurt |
| Add sparse-coding attention to LoRA | Llama-7B with LoRA · Commonsense reasoning benchmarks | accuracy: 75.6 -> 76.3 | Mixed |
| Add sparse-coding attention to OCL | OCL · 3D Shapes | MSE: 4.36 -> 4.31 | Helped |
| Add sparse-coding attention to OCL | OCL · BitMoji Faces | MSE: 4.77 -> 4.42 | Helped |
| Add sparse-coding attention to OCL | OCL · CLEVr Objects | MSE: 37.54 -> 36.23 | Helped |
| Add ReLU to both basis functions φ and ψ (ablation) | Sparse-coding attention · S-RAVEN | accuracy: 71.7 -> 73.6 | Helped |
| Add ReLU to φ only (ablation) | Sparse-coding attention · S-RAVEN | accuracy: 71.7 -> 72.3 | Helped |
| Add ReLU to ψ only (ablation) | Sparse-coding attention · S-RAVEN | accuracy: 71.7 -> 72.9 | Helped |
| Increase the soft-threshold ξ (ablation) | Sparse-coding attention · RAVEN | attention-map sparsity: 18.53 -> 99.38 | Helped |
Surprising findings
- The method applied to frozen Llama-7B with only about 128 new parameters improves average commonsense accuracy over the base model (70.5 to 71.9), though it does not beat LoRA or DoRA alone.
Limitations
- Evaluated only on small-scale tasks; application to large pre-trained models is unexplored.
- On commonsense reasoning, the method does not surpass LoRA or DoRA and had no extensive hyperparameter tuning.
Experimental setup
Datasets: S-RAVEN RAVEN synthetic compositional panel dataset commonsense reasoning benchmarks Im-promptu
Benchmarks: S-RAVEN RAVEN BoolQ PIQA HellaSwag WinoGrande ARC-c OBQA Im-promptu
Metrics: accuracy PSNR threshold coverage MSE sparsity
Baselines: standard Transformer HYLA LoRA DoRA Object-Centric Learner
Builds on: Transformer attention sparse coding lifting scheme HYLA LoRA DoRA
Closest papers by meaning
- Limits of Transformer Language Models on Learning to Compose Algorithms (NeurIPS 2024 · similarity 0.88)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.88)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.87)
- Does Data Scaling Lead to Visual Compositional Generalization? (ICML 2025 · similarity 0.87)
- Layer-Wise Representation Fusion for Compositional Generalization (AAAI 2024 · similarity 0.87)
- Provable Compositional Generalization for Object-Centric Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.