Research map / Probabilistic and scientific modeling / Molecules and proteins
VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQDNA, which replaces hand-crafted genome tokenization with a learned VQ-VAE codebook vocabulary, plus a hierarchical residual quantization (HRQ) variant, and evaluates on 32 genome datasets.
Problem
Hand-crafted tokenizers (k-mer, BPE) used in genome language models may not capture the most discriminative patterns from the limited four-nucleotide vocabulary, hurting sample efficiency and generalization.
Approach
Learn a genome vocabulary end-to-end as a vector-quantized codebook (VQ-VAE tokenizer), and extend it with hierarchical residual quantization whose codebooks grow in size across layers to enrich the vocabulary coarse-to-fine.
Key results
- VQDNA (HRQ) reaches average rank 1 across 32 datasets in the efficiency table, vs. DNABERT-2 rank 3 and NT-2500M rank 4, with 86+17M parameters versus 2537M for NT-2500M.
- VQDNA (HRQ) achieves 90.75 MCC on PD (all), 71.02 on CPD (all), and 78.10 on TFP-Human index 4, outperforming DNABERT-2 on most GUE tasks.
- On 5-species classification at 32k length, VQDNA (HRQ) reaches 99.46% top-1 accuracy vs. 99.28% for DNABERT-2 and 93.42% for HyenaDNA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical residual quantization to VQDNA (ablation) | VQDNA · Covid Variants Classification | F1 (%): 73.16 -> 74.32 | Helped |
| Add hierarchical residual quantization to VQDNA (ablation) | VQDNA · Splice Site Prediction (reconstructed) | MCC (%): 88.06 -> 89.53 | Helped |
| Add hierarchical residual quantization to VQDNA (ablation) | VQDNA · Promoter Detection (all) | MCC (%): 90.20 -> 90.75 | Helped |
| Add hierarchical residual quantization to VQDNA (ablation) | VQDNA · Species Classification (32k) | top-1 accuracy (%): 99.28 -> 99.46 | Helped |
| Use a 25% MLM masking ratio instead of 20% (ablation) | VQDNA · Epigenetic Marks Prediction (H3) | MCC (%): 78.3 -> 78.6 | Helped |
| Use a 25% MLM masking ratio instead of 20% (ablation) | VQDNA · Covid Variants Classification | F1 (%): 73.4 -> 73.2 | Hurt |
| Use a 25% MLM masking ratio instead of 20% (ablation) | VQDNA + HRQ · Covid Variants Classification | F1 (%): 74.2 -> 74.3 | Helped |
| Vary MLM masking ratio across 15%, 20%, 25%, and 30% (ablation) | VQDNA and VQDNA + HRQ · H3 and Covid Variants Classification | MCC (%) / F1 (%) | Mixed |
| Use a codebook size of 512 rather than other tested sizes (ablation) | VQDNA and HRQ · CVC token reconstruction and linear probing | reconstruction accuracy / linear probing accuracy | Helped |
| Vary codebook dimension (ablation) | VQDNA tokenizers · CVC token representation | linear probing accuracy | No clear effect |
Surprising findings
- A codebook size of 512 gave the best reconstruction/discrimination trade-off, and codebook dimension had less effect on representation quality.
- Masking ratio of 25% outperformed the 15% and 20% ratios common in prior models, which the authors attribute to richer contextual information in VQ tokens.
Limitations
- The genome vocabulary learning stage adds extra training cost compared with other genome language models.
- Model scale and pre-training data were limited by computational constraints, so scaling behavior is unexplored.
- Evaluated on classification/regression tasks only; generation tasks not studied.
Experimental setup
Datasets: Human genome (2.75B nucleotides) Multi-species genome (32.49B nucleotides, 135 species) GUE benchmark (28 datasets) Editing efficiency prediction (K562, Jurkat, H1) Species classification long-range dataset GISAID SARS-CoV-2 variants (Covid Variant Classification)
Benchmarks: GUE EEP species classification MUSE-style downstream tasks
Metrics: MCC F1 Accuracy Spearman correlation GFLOPs
Baselines: DNABERT Nucleotide Transformer (NT-500M-1000g) Nucleotide Transformer (NT-2500M-1000g) DNABERT-2 HyenaDNA
Builds on: VQ-VAE DNABERT-2 residual quantization ConvNeXt MaskGIT
Closest papers by meaning
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.87)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.86)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.86)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.85)
- HELM: Hierarchical Encoding for mRNA Language Modeling (ICLR 2025 · similarity 0.85)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.