Research map / Architectures and efficiency / Attention and transformers
Geometric Hyena Networks for Large-scale Equivariant Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Geometric Hyena, an SE(3)-equivariant long-convolutional network that captures global geometric context in sub-quadratic time via FFT-based vector and geometric long convolutions.
Problem
Modeling global geometric context while preserving SE(3) equivariance is computationally expensive: equivariant self-attention is quadratic in sequence length, while local message passing loses global information.
Approach
Extends the Hyena long-convolution operator to equivariant geometric sequences by introducing a vector long convolution (cross-product decomposed into six scalar FFT convolutions) and a geometric long convolution enabling scalar-vector interactions, combined with gating, key-value normalization, and EGNN projections with global context tokens.
Key results
- On Open Vaccine, G-Hyena reduces RMSE to 0.363 (backbone) and 0.339 (all-atom), about 12% and 15% better than the second-best global model.
- On protein MD, G-Hyena achieves MSE 1.80 (backbone) and 2.49 (all-atom), reducing error by 2% and 9% versus the second-best model.
- Processes 30k-token geometric context about 20x faster than the equivariant transformer and supports up to 2.7M tokens on a single A10G GPU (72x longer context within the same budget).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use G-Heyna instead of the strongest listed baseline | RNA property prediction · Open Vaccine Covid-19, backbone | RMSE: 0.400 -> 0.363 | Helped |
| Use G-Heyna instead of the strongest listed baseline | RNA property prediction · Ribonanza-2k, backbone | RMSE: 0.537 -> 0.529 | Helped |
| Use G-Heyna instead of the strongest listed baseline | RNA property prediction · Open Vaccine Covid-19, all-atom | RMSE: 0.393 -> 0.339 | Helped |
| Use G-Heyna instead of the strongest listed baseline | RNA property prediction · Ribonanza-2k, all-atom | RMSE: 0.592 -> 0.546 | Helped |
| Use G-Heyna instead of the strongest listed baseline | RNA switching-factor prediction · Tc-Ribo, backbone | RMSE: 0.550 -> 0.548 | Helped |
| Use G-Heyna instead of the strongest listed baseline | RNA switching-factor prediction · Tc-Ribo, all-atom | RMSE: 0.553 -> 0.548 | Helped |
| Use G-Heyna instead of the strongest listed baseline | Protein MD prediction · ProteinMD, backbone | MSE: 1.84 -> 1.80 | Helped |
| Use G-Heyna instead of the strongest listed baseline | Protein MD prediction · ProteinMD, all-atom | MSE: 2.72 -> 2.49 | Helped |
| use K gating instead of QK gating (ablation) | G-Heyna · Open Vaccine Covid-19, all-atom ablation | RMSE: 0.339 -> 0.345 | Hurt |
| remove gating (ablation) | G-Heyna · Open Vaccine Covid-19, all-atom ablation | RMSE: 0.339 -> 0.449 | Hurt |
| remove local context (ablation) | G-Heyna · Open Vaccine Covid-19, all-atom ablation | RMSE: 0.339 -> 0.450 | Hurt |
| remove global context (ablation) | G-Heyna · Open Vaccine Covid-19, all-atom ablation | RMSE: 0.339 -> 0.421 | Hurt |
Surprising findings
- Standard non-equivariant Hyena and Transformer trained with data augmentation perform competitively with global equivariant methods on the Tc-Ribo switching-factor task.
Limitations
- FFT-based convolution is not permutation equivariant (only cyclic shifts), so it requires a canonical ordering and is less suited to unordered point clouds.
- Evaluated only on ordered biomolecular geometric graphs with canonical ordering.
- Key-value normalization is required for stable training; without it the model yields NaN.
Experimental setup
Datasets: Open Vaccine Covid-19 Ribonanza-2k Tc-Ribo AdK protein MD trajectories (via MDAnalysis) geometric associative recall (synthetic)
Benchmarks: Open Vaccine Covid-19 Ribonanza-2k Tc-Ribo ProteinMD geometric associative recall
Metrics: RMSE MSE runtime (ms) peak GPU memory (GB)
Baselines: SchNet TFN EGNN FastEGNN LEFTNet TMD-ET Transformer Hyena VNT G-Transformer Equiformer
Builds on: Hyena EGNN Equiformer Vector Neuron Transformer
Closest papers by meaning
- DualEqui: A Dual-Space Hierarchical Equivariant Network for Large Biomolecules (NeurIPS 2025 · similarity 0.89)
- HAGO-Net: Hierarchical Geometric Message Passing for Molecular Representation Learning (AAAI 2024 · similarity 0.86)
- Generalist Equivariant Transformer Towards 3D Molecular Interaction Learning (ICML 2024 · similarity 0.86)
- Large Language-Geometry Model: When LLM meets Equivariance (ICML 2025 · similarity 0.86)
- Beyond Sequence: Impact of Geometric Context for RNA Property Prediction (ICLR 2025 · similarity 0.85)
- E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.