Research map / Architectures and efficiency / RNNs and normalization
Discrete-Valued Neural Communication
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer, Yoshua Bengio
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add discretized communication (DVNC) to transformer attention outputs, G=8 | Transformer · Sort-of-CLEVR binary | accuracy: 76.00 -> 88.00 | Improved |
| add DVNC to transformer attention outputs, G=8 | Transformer · Sort-of-CLEVR ternary | accuracy: 57.25 -> 62.67 | Improved |
| add DVNC to transformer attention outputs, G=8 | Transformer · Sort-of-CLEVR unary | accuracy: 97.75 -> 98.75 | No clear effect |
| use a single discretization head (G=1) instead of G=8 (ablation) | DVNC transformer · Sort-of-CLEVR binary | accuracy: 88.00 -> 80.50 | Hurt |
| increase heads from G=8 to G=16 (ablation) | DVNC transformer · Sort-of-CLEVR binary | accuracy: 88.00 -> 84.00 | Hurt |
| add DVNC to GNN edge communication | GNN · Atari OOD movement prediction (Alien, Pong, SpaceInvaders) | score: 0.1991 -> 0.2876 (Alien) | Improved |
| add DVNC to GNN edge communication | GNN · Atari OOD Boxing | score: 0.9228 -> 0.9502 | No clear effect |
| add DVNC to GNN edge communication | GNN · Atari OOD Berzerk | score: 0.6077 -> 0.6233 | No clear effect |
| discretize communication results rather than edge hidden states (ablation) | GNN with DVNC · 2D/3D shapes OOD | HITs | Improved |
| discretize attention communication results rather than recurrent updates or inputs (ablation) | RIMs with DVNC · Adding task OOD | OOD performance | Improved |
| use VQ-VAE nearest-neighbor discretization instead of Gumbel-Softmax (ablation) | RIMs DVNC · Adding task OOD | OOD performance | Improved |
Closest papers by meaning
- Sparse Communication via Mixed Distributions (ICLR 2022 · similarity 0.85)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.84)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.84)
- A compressive-expressive communication framework for compositional representations (NeurIPS 2025 · similarity 0.84)
- The impact of allocation strategies in subset learning on the expressive power of neural networks (ICLR 2025 · similarity 0.84)
- Codebook Features: Sparse and Discrete Interpretability for Neural Networks (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
