Research map / Large language models / Language, safety and interpretability
DNACHUNKER: Learnable Tokenization for DNA Language Models
Taewon Kim, Jihwan Shin, Hyomin Kim, Youngmok Jung, Jonghoon Lee, Won-Chul Lee, Sungsoo Ahn, Insu Han
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mask protection (forced boundaries around masked bases) (ablation) | DNAChunker 50M · Revised NT (hg38, linear probe) | overall MCC: 0.332 -> 0.390 | Improved |
| add masked residual gating in decoder (ablation) | DNAChunker 50M · Revised NT (hg38, linear probe) | overall MCC: 0.353 -> 0.390 | Improved |
| add ratio loss controlling compression (ablation) | DNAChunker 50M · Revised NT (hg38, linear probe) | overall MCC: 0.348 -> 0.390 | Improved |
| add ratio loss controlling compression (ablation) | DNAChunker 50M · Revised NT splice site | MCC: 0.123 -> 0.290 | Improved |
| replace 6-mer tokenization with learned chunking (ablation) | DNAChunker 50M · Revised NT (hg38, linear probe) | overall MCC: 0.347 -> 0.390 | Improved |
| replace BPE tokenization with learned chunking (ablation) | DNAChunker 50M · Revised NT (hg38, linear probe) | overall MCC: 0.375 -> 0.390 | Improved |
| pretrain on multispecies instead of human genome only (ablation) | DNAChunker 50M · Revised NT (linear probe) | overall MCC: 0.390 -> 0.448 | Improved |
| learned chunking vs BPE under multispecies pretraining (ablation) | DNAChunker 50M · Revised NT (linear probe) | overall MCC: +0.015 -> +0.073 lead over BPE | Improved |
| use learned chunking instead of BPE | DNAChunker · Motif fragmentation | mean tokens per motif: 2.47 -> 1.22 | Improved |
| use learned chunking instead of BPE | DNAChunker · Mutation robustness (ClinVar InDel benign) | tokenization similarity: 0.751 -> 0.851 | Improved |
| use learned chunking instead of BPE | DNAChunker · Mutation robustness (ClinVar SNV) | tokenization similarity: 0.999 -> 0.999 | No clear effect |
| use learned chunking instead of BPE or single-nucleotide tokens | DNAChunker 170M · 16k-bp forward FLOPs | FLOPs: BPE 1.58x vs DNAChunker; single-bp 17.6x | Improved |
Closest papers by meaning
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.89)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.88)
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.87)
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (NeurIPS 2024 · similarity 0.87)
- GeneZip: Region-Aware Compression for Long Context DNA Modeling (NeurIPS 2026 · similarity 0.87)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
