Research map / Architectures and efficiency / RNNs and normalization
Discrete Dictionary-based Decomposition Layer for Structured Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D3, a drop-in discrete key-value dictionary layer that generates structured TPR components (roles, fillers, unbinding operators) by sparse nearest-key lookup, improving systematic generalization of TPR-based models with few extra parameters.
Problem
TPR-based neural networks struggle to decompose unseen combinatorial inputs into structured role/filler/unbinding-operator representations, which undermines their symbolic operations and systematic generalization.
Approach
Each TPR component gets its own discrete learnable key-value dictionary; input queries retrieve top-k nearest keys (L2-normalized inner product) and aggregate the associated codebook values, with a residual feed-forward path, so decomposition maps inputs to pre-learned discrete symbolic features. Roles and unbinding operators share a dictionary, and the layer replaces the component-generation laye
Key results
- On the SAR task, FWM with D3 reaches nearly 100% accuracy on unseen combinations where FWM and AID fail.
- On sys-bAbI, TPR-RNN+D3 reduces the w/ sys diff word error rate from 5.61% (AID-style baseline) to 3.50% with 0.17M params, and FWM+D3 (w/ F) reaches 1.96% vs 2.85% for FWM.
- On sort-of-CLEVR, Linear Transformer+D3 (w/ F, D_code 256) reaches 99.0 Unary, 82.1 Binary, 68.8 Ternary vs 98.9/78.6/63.7 for the AID-enhanced baseline; on WikiText-103 test perplexity drops from 37.533 to 37.009.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add D3 | TPR-RNN · sys-bAbI w/ sys diff | word error rate: 5.61 -> 3.50 | Helped |
| Add D3 without applying it to fillers | FWM · sys-bAbI w/ sys diff | word error rate: 2.85 -> 2.58 | Helped |
| Add D3 and apply it to fillers | FWM · sys-bAbI w/ sys diff | word error rate: 2.85 -> 1.96 | Helped |
| Add D3 with D_code 256 and apply it to fillers | Linear Transformer · Sort-of-CLEVR Binary | accuracy: 75.5 -> 82.1 | Helped |
| Add D3 with D_code 256 and apply it to fillers | Linear Transformer · Sort-of-CLEVR Ternary | accuracy: 56.4 -> 68.8 | Helped |
| Add D3 with D_code 64 without applying it to fillers | Linear Transformer · WikiText-103 Test | perplexity: 37.533 -> 37.009 | Helped |
| Add D3 with D_code 32 and N_code 64 | FWM · SAR | accuracy: 44.90 -> 99.27 | Helped |
| Remove the codebook (ablation) | FWM + D3 · SAR | accuracy: 99.27 -> 89.02 | Hurt |
| Remove the codebook (ablation) | FWM + D3 · sys-bAbI w/ sys diff | word error rate: 1.96 -> 3.55 | Hurt |
| Use a single codebook element instead of multiple elements (ablation) | FWM + D3 · SAR | accuracy: 89.02 -> 89.10 | Hurt |
| Remove the residual connection (ablation) | D3 · SAR | accuracy | Hurt |
| Reduce D_code to 8 (ablation) | D3 · SAR | accuracy | Hurt |
Surprising findings
- Without the discrete codebook, the shared residual feed-forward networks alone already improve SAR accuracy (89.02% vs 44.90% for FWM), but do not reach near-perfect generalization.
- Applying D3 to fillers helps on sys-bAbI and sort-of-CLEVR but not on SAR and WikiText-103, so the best configuration is task-dependent.
Limitations
- Computational and memory overhead grows with dictionary capacity and number of TPR components.
- Choosing the w/ F or w/o F configuration per task is an extra burden without clear guidance.
- Sort-of-CLEVR results remain weaker than vanilla Transformer and Compositional Transformer on Binary and Ternary questions.
Experimental setup
Datasets: Systematic Associative Recall (SAR) sys-bAbI (from bAbI en-valid-10k) Sort-of-CLEVR WikiText-103
Benchmarks: SAR sys-bAbI Sort-of-CLEVR WikiText-103
Metrics: accuracy word error rate perplexity gap between w/o and w/ sys diff parameter count
Baselines: FWM TPR-RNN Linear Transformer AID Transformer Compositional Transformer Delta Network DAM STM
Builds on: Attention-based Iterative Decomposition (AID) Fast Weight Memory TPR-RNN Linear Transformer product key memory
Closest papers by meaning
- Attention-based Iterative Decomposition for Tensor Product Representation (ICLR 2024 · similarity 0.89)
- Mapping natural-language problems to formal-language solutions using structured neural representations (ICML 2020 · similarity 0.86)
- Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products (NeurIPS 2024 · similarity 0.85)
- Bridging Neural and Symbolic Representations with Transitional Dictionary Learning (ICLR 2024 · similarity 0.84)
- SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective (CVPR 2024 · similarity 0.83)
- Codebook Features: Sparse and Discrete Interpretability for Neural Networks (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.