Research map / Architectures and efficiency / RNNs and normalization
Improving Bilinear RNN with Closed-loop Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Comba, a bilinear RNN with scalar-plus-low-rank state transition and closed-loop state and output feedback, with a Triton chunk-wise parallel kernel, evaluated on language modeling at 340M/1.3B and vision tasks.
Problem
Efficient sequence models with Delta-rule memory correction (bilinear RNNs) lack a principled control view and trade off expressiveness against hardware-efficient chunk-wise parallel training.
Approach
Reframes Delta-rule bilinear RNNs as closed-loop control systems: uses an SPLR state transition (scalar decay plus one rank-one key term) with state feedback correction on values and an output feedback correction q - d*k, with a chunk-wise WY/UT parallel implementation.
Key results
- Comba-splr at 1.3B reaches 51.78 average commonsense accuracy vs 49.58 for Gated DeltaNet and a lower WikiText perplexity (16.19 vs 17.14).
- Comba shows nearly 40% forward-pass speedup over Gated DeltaNet in operator benchmarks.
- On GOT-10k/LaSOT tracking, Comba reaches AO 0.718 and SR0.75 0.688, exceeding softmax attention by 1.4% and 1.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPLR instead of IPLR state transition | Comba · SlimPajama 340M and 1.3B commonsense reasoning and recall tasks | average accuracy: 43.06 -> 43.36; 50.77 -> 51.78; 28.10 -> 30.40; 32.67 -> 36.37 | Helped |
| Add output correction <q, dk> (ablation) | Comba · SlimPajama 340M and 1.3B commonsense reasoning and recall tasks | average accuracy: 42.45 -> 43.36; 28.21 -> 30.40; 49.72 -> 51.78; 34.49 -> 36.37 | Helped |
| Add state feedback scaling tilde-beta = b⊙beta (ablation) | Comba · SlimPajama 340M commonsense reasoning | average accuracy: 43.13 -> 43.36 | Helped |
| Use alpha near 1 initialization (ablation) | Comba · SlimPajama 340M commonsense reasoning and LongBench 10K | average score: 42.68 -> 43.36; 18.85 -> 19.16 | Helped |
| Add the output gate (ablation) | Comba · SlimPajama 340M commonsense reasoning | average accuracy: 42.64 -> 43.36 | Helped |
| Initialize d to 1 instead of the small-model default (ablation) | Comba · SlimPajama 340M commonsense reasoning | average accuracy: 43.26 -> 43.36 | No clear effect |
| Initialize d to 1 (ablation) | Comba · SlimPajama 1.3B commonsense reasoning | average accuracy: 51.78 -> 51.91 | Helped |
| Use the Mamba-like architecture (ablation) | Comba · SlimPajama 340M commonsense reasoning | average accuracy: 43.36 -> 41.64 | Hurt |
| Use Comba-SPLR instead of Gated-DeltaNet | Comba · LongBench 10K | average score: 18.61 -> 19.16 | Mixed |
| Use the Comba chunk-wise operator instead of Gated-DeltaNet | Comba · Triton-Testing-Benchmark | forward speed: 40% | Helped |
Surprising findings
- The IPLR variant (Comba-iplr) achieves lower training loss than the SPLR variant despite SPLR scoring higher on downstream tasks.
- Initializing the output correction d to 0.02 works for 340M models while d=1 is best for 1.3B, showing scale-dependent sensitivity.
Limitations
- Experiments not extended to larger models such as 2.7B due to limited compute.
- Titans, Lattice, and MIRAS were not open-sourced, so direct comparison was not possible.
- Lags Gated DeltaNet on summarization and code long-context tasks; no error bars or statistical significance reported.
Experimental setup
Datasets: SlimPajama LongBench MQAR synthetic GOT-10k LaSOT ImageNet
Benchmarks: WikiText perplexity LAMBADA ARC HellaSwag PIQA WinoGrande FDA SWDE SQuAD NQ TriviaQA Drop
Metrics: perplexity accuracy average accuracy AO SR0.75 Top-1 accuracy FLOPs
Baselines: Transformer++ RetNet GLA Mamba RWKV7 Gated DeltaNet Comba-iplr DeiT Vim Agent Attention
Builds on: DeltaNet Gated DeltaNet RWKV-7 Mamba2 flash-linear-attention WY representation
Closest papers by meaning
- Resurrecting Recurrent Neural Networks for Long Sequences (ICML 2023 · similarity 0.88)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.88)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.