Research map / Architectures and efficiency / Attention and transformers
Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a matrix-mixer framework with sequence alignment as a key property, and Hydra, a bidirectional Mamba/SSD extension parameterized as a quasiseparable matrix mixer that beats BERT on GLUE and ViT on ImageNet-1K.
Problem
Sequence models (Transformers, SSMs, MLP-Mixer, FNet, etc.) are ad hoc and lack a unified view of their matrix structure; SSMs like Mamba are causal, and prior bidirectional SSM extensions are heuristic.
Approach
Views sequence mixers as structured L x L mixer matrices, defines sequence alignment (data-dependence and extendability) as a key axis, and makes bidirectional SSMs principled by using quasiseparable matrices, which decompose into two semiseparable (SSD) products via flip and shift plus a diagonal term.
Key results
- Hydra achieves a GLUE average of 84.3 versus BERT's 83.5 (+0.8) with 112M parameters, and a C4 masked-LM accuracy of 69.1% versus BERT's 67.3%.
- On ImageNet-1K, Hydra reaches 81.0% Top-1 accuracy, outperforming ViT-Base by 2.2 points.
- In the matrix-mixer ablation, data-dependent quasiseparable mixers reach GLUE avg 79.7 versus 75.1 for the data-independent quasiseparable variant; the bidirectional quasiseparable mixer beats Add/Mult/Concat Mamba variants (81.7 vs 80.6-81.1 GLUE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BERT with Hydra | BERT-style masked language model · GLUE | GLUE average: +0.8 points | Helped |
| replace ViT sequence mixer with Hydra | ViT-Base · ImageNet-1K | Top-1 accuracy: 2.2% | Helped |
| use quasiseparable bidirectional mixer instead of addition-based bidirectionality | Mamba · GLUE | GLUE average: 80.6 -> 81.7 | Helped |
| use quasiseparable bidirectional mixer instead of Hadamard-product bidirectionality | Mamba · GLUE | GLUE average: 79.6 -> 81.7 | Helped |
| use quasiseparable bidirectional mixer instead of concatenation | Mamba · GLUE | GLUE average: 81.1 -> 81.7 | Helped |
| add sequence alignment and input-dependent parameterization to quasiseparable mixer (ablation) | Quasiseparable mixer in the SSD block · GLUE | GLUE average: 75.1 -> 79.7 | Helped |
| add sequence alignment and input-dependent parameterization to Attention mixer (ablation) | Attention mixer in the SSD block · GLUE | GLUE average: 71.9 -> 78.8 | Helped |
| add sequence alignment and input-dependent parameterization to Low-Rank mixer (ablation) | Low-Rank mixer in the SSD block · GLUE | GLUE average: 74.9 -> 78.4 | Helped |
| add sequence alignment and input-dependent parameterization to Cauchy mixer (ablation) | Cauchy mixer in the SSD block · GLUE | GLUE average: 74.2 -> 78.2 | Helped |
| add sequence alignment and input-dependent parameterization to Toeplitz mixer (ablation) | Toeplitz mixer in the SSD block · GLUE | GLUE average: 75.8 -> 77.7 | Helped |
| add sequence alignment and input-dependent parameterization to Vandermonde mixer (ablation) | Vandermonde mixer in the SSD block · GLUE | GLUE average: 70.8 -> 76.0 | Helped |
| replace bidirectional Quasi mixer with unidirectional Mamba | Mamba sequence model · GLUE | GLUE average: 81.7 -> 77.7 | Hurt |
Surprising findings
- Unidirectional Mamba trails bidirectional variants by about 4.0 GLUE points, and a naive Mamba with CLS replaced by average pooling is the weakest in the comparison.
- Sequence-aligned Cauchy mixers, a newly proposed structure, perform nearly on par with low-rank (linear attention) mixers on GLUE.
Limitations
- Structured matrix mixers are less expressive than dense softmax attention, especially for memorization-heavy tasks.
- Structured matrix algorithms may be hardware-unfriendly compared with pure matmul-based Transformers.
Experimental setup
Datasets: C4 GLUE ImageNet-1K
Benchmarks: GLUE C4 validation (masked LM) ImageNet-1K
Metrics: GLUE average accuracy C4 cross-entropy loss masked token accuracy Top-1 accuracy Top-5 accuracy
Baselines: BERT ViT MLP-Mixer FNet M2 Mamba S4ND Hyena Add/Mult/Concat bidirectional Mamba
Builds on: Mamba SSD (Mamba-2) S4 Linear Attention M2
Closest papers by meaning
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.86)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.85)
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.85)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.85)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.85)
- EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.