Research map / Architectures and efficiency / Attention and transformers
Leverage the Explainability of Transformer Models to Improve the DNA 5-Methylcytosine Identification (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EA-5mC, an ensemble of two fine-tuned MuLan-Methyl transformer encoders that adds a custom layer built from ranked self-attention weights to classify 5-methylcytosine status in DNA sequences.
Problem
Identifying 5-methylcytosine (5mC) methylation status from DNA sequence with machine learning, under balanced and imbalanced (1:10) data distributions.
Approach
Treat the rank positions of attention weights assigned to [CLS] for the target cytosine tokens, aggregated across layers, as an explicit feature layer added to the averaged last hidden states of two fine-tuned transformer encoders (DistilBERT and ELECTRA variants of MuLan-Methyl).
Key results
- On the balanced 1:1 test set, EA-5mC reached accuracy 0.9463, AUC 0.9674, F1 0.9489, versus baseline 0.9461, 0.9669, 0.9487.
- On the 1:10 imbalanced test set, EA-5mC reached accuracy 0.9033 and F1 0.6514 versus baseline 0.9025 and 0.6502; baseline AUC was slightly higher (0.9661 vs 0.9647).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA balanced test (1:1) | Accuracy: 0.9461 -> 0.9463 | Helped |
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA balanced test (1:1) | AUC: 0.9669 -> 0.9674 | Helped |
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA balanced test (1:1) | F1-score: 0.9487 -> 0.9489 | Helped |
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA imbalanced test (1:10) | Accuracy: 0.9025 -> 0.9033 | Helped |
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA imbalanced test (1:10) | AUC: 0.9661 -> 0.9647 | Hurt |
| Add an attention-rank custom layer to the naive ensemble | MuLan-Methyl-DistiBERT + MuLan-Methyl-ELECTRA · MR-DNA imbalanced test (1:10) | F1-score: 0.6502 -> 0.6514 | Helped |
Surprising findings
- On the 1:10 imbalanced set, the baseline has a slightly higher AUC than EA-5mC, so the proposed framework does not win on every metric.
Limitations
- Gains over the naive ensemble baseline are very small (about 0.0002 to 0.001 on most metrics).
- Only two encoders and a single dataset (MR-DNA) are evaluated; no significance testing reported.
- The custom attention-rank layer construction is acknowledged as needing optimization.
Experimental setup
Datasets: MR-DNA
Benchmarks: MR-DNA test split (balanced 1:1 and 1:10)
Metrics: Accuracy AUC F1-score
Baselines: naive ensemble of the two fine-tuned models
Builds on: MuLan-Methyl BERT DNABERT
Closest papers by meaning
- Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis (NeurIPS 2025 · similarity 0.81)
- Revisiting Convolution Architecture in the Realm of DNA Foundation Models (ICLR 2025 · similarity 0.81)
- Multi-modal Transfer Learning between Biological Foundation Models (NeurIPS 2024 · similarity 0.81)
- What DNA Foundation Models Learn Beyond Sequence Composition (NeurIPS 2026 · similarity 0.81)
- Molecule Design by Latent Prompt Transformer (NeurIPS 2024 · similarity 0.80)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.