Research map / Architectures and efficiency / Attention and transformers

GeneZip: Region-Aware Compression for Long Context DNA Modeling

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GeneZip, a region-aware DNA compression module built on H-Net dynamic chunking with a Region-Aware Ratio loss from gene annotations and bounded routing, enabling ~137.6 bp/token compression and efficient long-context DNA pretraining.

Problem

Long-context DNA models are bottlenecked by token-mixing cost over very long sequences, and existing compression (fixed downsampling or content-only learned chunking) does not allocate representational budget according to genomic region structure.

Approach

Extends H-Net dynamic chunking with a Region-Aware Ratio (RAR) loss that sets region-specific base-pairs-per-token targets from gene-structure annotations during training only, plus bounded routing (token floor/ceiling) for stable training; inference needs no annotations.

Key results

• GeneZip-70M reaches 137.6 BPT with validation PPL 2.7259 vs 2.4115 for H-Net-3BPT reference and 2.7930 for H-Net-128BPT, the best among encoder-compression methods at 12.8K.
• GeneZip-70M achieves the best average rank (2.0) across four DNALongBench tasks, with eQTL AUROC 0.820 and ETGP AUPRC 0.446.
• GeneZip-70M fine-tunes the eQTL Artery_Tibial task in 50 minutes versus 2520 minutes for JanusDNA (50.4x faster).

Surprising findings

• On TISP, the uncompressed Mamba2 baseline (0.294) outperforms GeneZip-70M (0.279), and JanusDNA nearly fails (0.002).
• H-Net compresses tandem repeats less than the matched background (0.79 in human), which GeneZip does not do.

Limitations

• Region-to-BPT multipliers are chosen per downstream task by validation, so task-specific priors are needed for best results.
• Several baselines were pretrained on different corpora, so comparisons are not fully controlled.
• RSAP task could not be reproduced and was excluded.

Experimental setup

Datasets: GENCODE human corpus (gencode_human_12.8k, gencode_human_128k) DNALongBench CMP, eQTL, ETGP, TISP RepeatMasker TE and TRF tandem repeat masks (post hoc)

Benchmarks: DNALongBench contact map prediction DNALongBench eQTL DNALongBench enhancer-target gene prediction DNALongBench transcription initiation signal prediction GENCODE region-wise validation perplexity

Metrics: perplexity bits per token compression ratio (BPT) SCC Pearson correlation AUROC AUPRC average rank fine-tuning time

Baselines: Mamba2 U-Net H-Net-128BPT H-Net-3BPT HyenaDNA Caduceus-Ph Caduceus-PS JanusDNA NTv3-100M-pre Evo2-1B

Builds on: H-Net Mamba2 DNALongBench

Closest papers by meaning

• DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.85)
• Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis (NeurIPS 2025 · similarity 0.85)
• JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.85)
• Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.85)
• Genomics Data Lossless Compression with (S, K)-Mer Encoding and Deep Neural Networks (AAAI 2025 · similarity 0.84)
• BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.84)