Research map / Architectures and efficiency / RNNs and normalization
Attention-based Iterative Decomposition for Tensor Product Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Attention-based Iterative Decomposition (AID), a slot-attention-style module with learnable routing that improves the role/filler decomposition of TPR-based sequence models, evaluated on synthetic and QA tasks.
Problem
TPR-based neuro-symbolic models learn incomplete decomposition of sequential inputs into roles and fillers, which degrades systematic generalization on unseen compositions.
Approach
An iterative competitive attention module (AID) that binds input features to TPR components, with each initial component routed to a specific TPR symbol (role, filler, or unbinding operator) and context-dependent initialization, drop-in for any TPR-based model.
Key results
- On sys-bAbI, FWM with AID reduces mean word error rate on the w/ sys diff setting from 2.85% to 1.21%.
- On Sort-of-CLEVR, Linear Transformer with AID raises unary accuracy from 69.3% to 98.9%.
- On WikiText-103, AID lowers test perplexity of the Linear Transformer from 37.533 to 37.151 and of Delta Network from 36.659 to 36.253.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AID to FWM | FWM · sys-bAbI w/ sys diff | word error rate: 2.85 -> 1.21 | Helped |
| Add AID to Linear Transformer | Linear Transformer · Sort-of-CLEVR Unary | accuracy: 69.3 -> 98.9 | Helped |
| Add AID to Linear Transformer | Linear Transformer · Sort-of-CLEVR Binary | accuracy: 75.5 -> 78.6 | Helped |
| Add AID to Linear Transformer | Linear Transformer · Sort-of-CLEVR Ternary | accuracy: 56.4 -> 63.7 | Helped |
| Add AID to Linear Transformer | Linear Transformer · WikiText-103 test | perplexity: 37.533 -> 37.151 | Helped |
| Add AID to Delta Network | Delta Network · WikiText-103 test | perplexity: 36.659 -> 36.253 | Helped |
| Add a residual connection to the query and concatenate after the iterative process (ablation) | AID · sys-bAbI and SAR | performance | Helped |
| Concatenate after the iterative process (ablation) | AID · SAR | performance | Hurt |
| Use ELU activation in dot-product attention (ablation) | AID · SAR and sys-bAbI | performance | Helped |
| Set the number of input features to 3 (ablation) | AID with FWM · SAR, p=0.0 | accuracy | Helped |
| Increase the number of decomposition iterations up to the training-time setting (ablation) | AID · SAR, p=0.0 | accuracy | Helped |
| Apply AID at the first rather than the fourth layer (ablation) | Linear Transformer · WikiText-103 test | perplexity: 37.425 -> 37.151 | Helped |
Surprising findings
- Concatenation technique gave unstable and lower performance on the SAR task in the ablation.
- Simply increasing baseline parameters (e.g., FWM with D_LSTM 512) did not match AID's gains on sys-bAbI.
Limitations
- Gains on sys-bAbI remain a gap between w/o and w/ systematic difference settings for some models (e.g., TPR-RNN+AID gap 4.92).
- Evaluation limited to synthetic and small-scale QA/VQA tasks plus WikiText-103 perplexity; no large pretrained LLM experiments.
Experimental setup
Datasets: bAbI Sort-of-CLEVR WikiText-103 Systematic Associative Recall (synthetic, newly designed)
Benchmarks: Systematic Associative Recall sys-bAbI Sort-of-CLEVR WikiText-103
Metrics: test accuracy word error rate perplexity DCI disentanglement/completeness/informativeness cosine similarity
Baselines: TPR-RNN FWM Linear Transformer Transformer-XL DAM STM DNC Delta Network
Builds on: Tensor Product Representation Slot Attention TPR-RNN Fast Weight Memory Linear Transformer
Closest papers by meaning
- Discrete Dictionary-based Decomposition Layer for Structured Representation Learning (NeurIPS 2024 · similarity 0.89)
- Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products (NeurIPS 2024 · similarity 0.87)
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.86)
- SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective (CVPR 2024 · similarity 0.86)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.84)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.