Research map / Architectures and efficiency / Pruning and efficient tuning
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GERM, a DNABERT-2-style genomic foundation model whose attention is replaced by an outlier-free Softmax1 (Hopfield-inspired) layer, plus GERM-T, a small-step continual-learning variant that converts existing checkpoints to reduce outliers and improve quantization and LoRA adaptation.
Problem
Genomic foundation models such as DNABERT-2 are costly to adapt and degrade badly under low-rank adaptation and post-training quantization because of attention outliers, and retraining outlier-free models from scratch is expensive.
Approach
Replace the vanilla softmax attention with an outlier-free Softmax_1 (associative-memory / Hopfield-derived) layer during pretraining and fine-tuning, and reduce outliers in existing checkpoints via small-step continual learning with the outlier-free structure.
Key results
- GERM reduces average performance drop under 8-bit SmoothQuant quantization from 43.81% (DNABERT-2) to 4.82%, and under W4A4 OmniQuant to 17.17% vs 94.78%.
- GERM reduces average kurtosis by 92.14% and maximum infinity norm by 82.77% across 27 datasets, per the paper.
- GERM-T (40K continual steps) reduces average kurtosis by about 7.20% and max infinity norm by 53.78% relative to DNABERT-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla softmax attention with outlier-free Softmax_1 (GERM) | DNABERT-2 · GUE PTQ and LoRA | MCC / avg performance drop: LoRA drop 13.87% -> 4.12%; W4A4 OmniQuant drop 94.78% -> 17.17% | Helped |
| Replace vanilla softmax attention with outlier-free Softmax_1 (GERM) | DNABERT-2 · GUE pretrained model outlier metrics | avg kurtosis / max inf norm: avg kurtosis 270.90 -> 21.29; max inf norm 61.64 -> 10.62 | Helped |
| Small-step continual learning (40K steps) with outlier-free attention (GERM-T) | DNABERT-2 · GUE PTQ 6W/6A Outlier Suppression and LoRA | avg performance drop: LoRA drop 13.87% -> 6.23%; max inf norm 61.41 -> 28.49 (full) | Helped |
| Longer continual learning (100K steps) vs 40K for GERM-T (ablation) | GERM-T outlier-free continual learning · GUE PTQ 8W/8A SmoothQuant | avg performance drop: 2.96% vs 3.00% (GERM-T 40K) | No clear effect |
| Continual learning steps 20K vs 40K/100K (ablation) | GERM-T · GUE LoRA | avg performance drop: Out20k 7.53% vs GERM-T 6.24% vs Out100k 6.52% (LoRA) | Mixed |
| Outlier-free attention with LoRA rank 8 vs vanilla (ablation) | DNABERT-2 / GERM · GUE LoRA rank ablation | MCC: 52.87 -> 57.27 MCC | Helped |
Surprising findings
- GERM-T's kurtosis reduction is minimal compared with GERM's, and it shows a notable drop at W4A4 quantization.
- Full-rank fine-tuned GERM-T at 100K continual steps performs better but shows a larger LoRA performance drop than the 40K variant.
Limitations
- GERM-T shows significant performance drops at 4-bit quantization and under QLoRA/LoftQ.
- Outlier reduction with continual learning is limited in average kurtosis, and retraining from scratch is still needed for full outlier removal.
- Evaluation focuses on classification tasks of the GUE benchmark and a 117M-parameter model.
Experimental setup
Datasets: GUE benchmark (27 datasets, 7 tasks, 4 species) Nucleotide Transformer benchmark
Benchmarks: GUE Nucleotide Transformer benchmark
Metrics: MCC average performance drop average kurtosis maximum infinity norm
Baselines: DNABERT-2 SmoothQuant Outlier Suppression OmniQuant LoRA QLoRA LoftQ clipped softmax gated attention
Builds on: DNABERT-2 Outlier-efficient Hopfield layers (OutEffHop) LoRA SmoothQuant OmniQuant
Closest papers by meaning
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.85)
- VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling (ICML 2024 · similarity 0.84)
- Genomics Data Lossless Compression with (S, K)-Mer Encoding and Deep Neural Networks (AAAI 2025 · similarity 0.83)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.83)
- What DNA Foundation Models Learn Beyond Sequence Composition (NeurIPS 2026 · similarity 0.83)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.