AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

John Cooper, Ilias Diakonikolas, Mingchen Ma, Frederic Sala

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
put SSM layer before the Transformer layer in a 2-layer hybrid (ablation)hybrid on Selective Copy (~2000 params) · Selective Copyaccuracy: 0.433 -> 0.999Improved
use hybrid instead of pure TF or SSM at ~2000 params2-layer models · Selective Copyaccuracy: 0.352 / 0.305 -> 0.999Improved
use hybrid instead of pure TF or SSM at ~6000 params2-layer models · Multi-Key Associative Recallaccuracy: 0.230 / 0.356 -> 0.990Improved
put Transformer layer before SSM layer in the hybrid (ablation)hybrid on Multi-Key Associative Recall (~6000 params) · Multi-Key Associative Recallaccuracy: 0.990 -> 0.286Hurt
use hybrid instead of pure models on associative recall with decoding (3 layers, ~1M params)3-layer models · Associative Recall with Decodingaccuracy: <0.4 -> >0.5Improved
use hybrid instead of SSM or Transformer for Needle-in-a-Haystack at small scale2-layer models, full-context attention · Needle in a HaystackaccuracyImproved
use hybrid instead of pure Transformer for longer test sequences~100M-param models trained on lengths 20-50 · Associative Recall with Decoding length generalizationaccuracy: within 2% on short sequences -> around 10% gap on longImproved
use hybrid instead of SSM/TF under shifted training bit proportions (train 0.3)12-layer models · Associative Recall with Decoding OOD, test proportion 0.2accuracy: 0.17 / 0.64 -> 0.74Improved
use hybrid with high training bit proportion 0.912-layer models · Associative Recall with Decoding OOD, test proportion 0.2accuracy: 0.86 (SSM) vs 0.80 (hybrid)Hurt
increase model size from ~2000 to ~12000 params for pure Transformer (ablation)pure TF on Selective Copy · Selective Copyaccuracy: 0.352 -> 0.923Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy