Research map / Probabilistic and scientific modeling / Molecules and proteins

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling

NeurIPS 2026 · Probabilistic and scientific modeling › Molecules and proteins · approach: Biological sequence transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ZEST, an evolution-informed tokenizer whose vocabulary of conserved MSA-derived zones compresses proteins ~4 residues per token, plus trie-dropout regularization, and LEMON, a 200M-parameter contrastively fine-tuned protein language model for remote homology detection.

Problem

Standard residue-level protein language models scale poorly and fail to exploit evolutionary conservation at the tokenization stage, leaving remote homology (fold-level) detection from sequence alone hard.

Approach

Build a vocabulary of multi-residue conserved zones from HMM consensus sequences (TEDLH), organize them in a trie, and apply trie-dropout to stochastically decompose long tokens during training, then train a compact encoder with MLM and hierarchical contrastive learning on CATH labels.

Key results

• LEMON (200M) reports mean AUROC 0.86 and mAP 0.31 across CATH S20, SCOPe and SCOP benchmarks, outperforming PLMs from 650M to 3B parameters.
• On SCOP fold-level retrieval, LEMON ranks first at all six identity thresholds, from 0.875 (th10) to 0.904 (th95).
• In a tokenizer ablation, ZEST (32K) reached ROC-AUC 0.739 vs 0.729 for BPE and 0.708 for character-level tokenization on SCOPe 10%-identity retrieval.

Surprising findings

• Character-level tokenization gave the lowest accuracy but BPE trained faster than char, with precision trade-off giving diminishing returns.
• LEMON underperforms ESM models on finer-grained sequence-level property prediction, as acknowledged in the limitations.

Limitations

• Underperforms ESM-family models on finer-grained sequence-level property prediction tasks.
• Benchmark performance is evaluated on ranking-based retrieval; GO MF results are essentially tied with ProtTucker, showing limited gains on functional tasks.
• Vocabulary is derived from TEDLH HMM consensus sequences, so coverage depends on the domain annotation source.

Experimental setup

Datasets: UniRef90 TEDLH TED CATH SCOP SCOPe CATH S20 CIRPIN SCOPe40 ProteInfer SwissProt EC test set MZSGO CATH-17287

Benchmarks: SCOP SCOPe CATH S20 CIRPIN SCOPe40 ProteInfer EC MZSGO GO MF CATH-17287 domain boundary

Metrics: AUROC mAP AUPRC F-max AUPR NDO NRes

Baselines: ESM-2 (650M) ESM-2 (3B) ProtTrans-T5-XL SaProt ProstT5 ProtTucker Ankh-Base Ankh-Large PLMSearch HHblits

Builds on: BPE-dropout ProtTucker ESM-2 HHblits ALiBi

Closest papers by meaning

• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.90)
• Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.89)
• Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.88)
• Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment (NeurIPS 2025 · similarity 0.88)
• ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (NeurIPS 2024 · similarity 0.88)
• SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.88)