Research map / Architectures and efficiency / RNNs and normalization
Trellis Networks for Sequence Modeling
Shaojie Bai, J. Zico Kolter, Vladlen Koltun
This paper tried a common change: Add a reconstruction / auxiliary loss
add auxiliary losses · Word-level PTB test · perplexity: 57.99 -> 56.97 Improved
What 987 accepted papers report for this change
- 882 improved
- 41 no clear effect
- 20 mixed
- 44 hurt
- OPTION: Optimal Transport–Guided Flow Matching for Incomplete and Unaligned Multi-View Clustering ICML 2026 · add reconstruction loss L_rec · Scene-15 (aligned, 70% missing) · ACC: 32.4 -> 37.5Improved
- Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR ICML 2026 · add R-probe auxiliary loss · OCRBench · score: 714.0 -> 721.0Improved
- Rethinking Time-Series Imputation as Conditional Inference along Temporal Evolution ICML 2026 · add reconstruction loss on observed entries (joint vs missing-only supervision) · ETTh1 10% missing · Masked MAE: 0.2002 -> 0.1658Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TrellisNet-MoS (1.4x larger) | Word-level PTB · Word-level PTB test | perplexity: 56.97 -> 54.19 | Improved |
| use TrellisNet instead of Relational Memory Core | Word-level WT103 · Word-level WT103 test | perplexity: 31.6 -> 29.19 | Improved |
| use TrellisNet instead of AWD-LSTM | Character-level PTB · Character-level PTB test | bits-per-character: 1.175 -> 1.158 | Improved |
| use TrellisNet instead of IndRNN | Sequential MNIST · Sequential MNIST | accuracy: 99.0 -> 99.20 | Improved |
| use TrellisNet instead of Transformer | Permuted MNIST · Permuted MNIST | accuracy: 97.9 -> 98.13 | Improved |
| use TrellisNet instead of r-LSTM with auxiliary loss | Sequential CIFAR-10 · Sequential CIFAR-10 | accuracy: 72.2 -> 73.42 | Improved |
| add hidden (VD-based) dropout (ablation) | TrellisNet · Word-level PTB test | perplexity: 64.69 -> 56.97 | Improved |
| add weight dropout (ablation) | TrellisNet · Word-level PTB test | perplexity: 63.82 -> 56.97 | Improved |
| add auxiliary losses (ablation) | TrellisNet · Word-level PTB test | perplexity: 57.99 -> 56.97 | Improved |
| add long-sequence parallelism (ablation) | TrellisNet · Word-level PTB test | perplexity: 57.35 -> 56.97 | Improved |
| use a dense kernel instead of mixed group convolution (ablation) | TrellisNet · Word-level PTB test | perplexity: 59.18 -> 56.97 | Improved |
| inject input at every layer rather than every 2 layers (ablation) | TrellisNet · Word-level PTB test | perplexity: 57.44 -> 56.97 | Improved |
Closest papers by meaning
- Convolutional Sequence to Sequence Learning (ICML 2017 · similarity 0.88)
- Language Modeling with Gated Convolutional Networks (ICML 2017 · similarity 0.87)
- Deep Equilibrium Models (NeurIPS 2019 · similarity 0.87)
- Quasi-Recurrent Neural Networks (ICLR 2017 · similarity 0.87)
- Hierarchically Gated Recurrent Neural Network for Sequence Modeling (NeurIPS 2023 · similarity 0.86)
- Universal Transformers (ICLR 2019 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
