Research map / Architectures and efficiency / RNNs and normalization
Protein Structure Tokenization: Benchmarking and New Recipe
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StructTokenBench, a unified framework evaluating protein structure tokenizers on downstream effectiveness, sensitivity, distinctiveness, and codebook utilization, and proposes a VQ-VAE tokenizer (with codebook reparameterization and Pareto-optimal codebook configuration) that improves over ESM3.
Problem
Protein structure tokenizers (PSTs) are used for structure language modeling and multimodal models, but their capabilities and limitations are poorly understood due to no unified evaluation framework; VQ-VAE-based PSTs also suffer from codebook under-utilization.
Approach
A four-perspective benchmark focused on fine-grained local substructure quality, plus a VQ-VAE tokenizer that reparameterizes the codebook as a learnable linear transform of a fixed orthogonal basis so all codes receive gradient updates, and selects codebook size/dimension under a fixed capacity K×D.
Key results
- The proposed model achieves a +6.31% average relative improvement over ESM3 across 24 supervised task splits.
- Sensitivity on Apo Holo and Fold Switching improves by +12.83% average relative gain over ESM3, and codebook utilization rate on CASP14 rises from 27.60% (ESM3) to 64.45%.
- Inverse-folding-based PSTs lead downstream effectiveness (ProteinMPNN/MIF average AUROC 75.92/79.82), while VQ-VAE-based ESM3 is best among VQ-VAE methods on most tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed tokenizer instead of ESM3 | Protein structure tokenization · Downstream tasks (24 splits) | Average relative performance: +6.31% | Helped |
| use the proposed tokenizer instead of ESM3 | Protein structure tokenization · Apo Holo and Fold Switching | Sensitivity: +12.83% | Helped |
| use codebook reparameterization and Pareto-optimal codebook configuration | ESM3-style VQ-VAE tokenizer · CASP14 and CAMEO | UR%: +124.03% | Helped |
| add amino acid tokens to structural tokens (ablation) | Protein Structure Tokenization: Benchmarking and New Recipe · Downstream tasks (24 splits) | AvgDiff%: 1.33 | Helped |
| use continuous rather than discrete structural representations (ablation) | VQ-VAE-based PSTs · Supervised downstream tasks | Task performance | Mixed |
| add amino acid sequence information to structural representations (ablation) | VQ-VAE-based PSTs · Supervised downstream tasks | Task performance | Helped |
| add noise to structural representations (ablation) | Protein structure tokenizers · Supervised downstream tasks | Task performance | Hurt |
| increase noise in structural representations (ablation) | Protein structure tokenizers · Supervised downstream tasks | Performance differences between methods: up to 90% | No clear effect |
| increase codebook size to K=2^14 (ablation) | VQ-VAE-based PST · Structure reconstruction | Reconstruction quality | Helped |
| increase codebook size beyond 2^11 (ablation) | VQ-VAE-based PST · Codebook utilization | Utilization | Hurt |
| use extreme codebook sizes instead of moderate sizes in [2^8, 2^10] (ablation) | VQ-VAE-based PST · Supervised downstream tasks | Task performance | Hurt |
| increase training data and compute (ablation) | VQ-VAE-based PST · Pretraining | Reconstruction loss | Helped |
Surprising findings
- Reconstruction quality does not consistently correlate with codebook quality; the best reconstruction at K=2^14 coincides with poor utilization.
- Up to 70% of ESM3's 4096 codes remain inactive during inference.
Limitations
- Evaluation focuses on backbone-only structure inputs; all-atom methods like Cheap are outside the scope.
- Pre-training uses only a 10% subset of PDB, so scale relative to ESM3's pretraining data is limited.
- Downstream tasks use a 2-layer MLP probe with fixed representations, which may not reflect fine-tuned performance.
Experimental setup
Datasets: PDB ATLAS InterPro BioLIP2 ProteinShake ProteinGLUE TAPE Fold Switching Apo Holo CAMEO CASP14
Benchmarks: StructTokenBench BindInt BindBio BindShake CatInt CatBio Con Rep Ept FlexRMSF FlexBFactor FlexNEQ
Metrics: AUROC Spearman's rho Macro F1 PCC cosine similarity UR perplexity MUV
Baselines: FoldSeek ProTokens ESM3 ProteinMPNN MIF
Builds on: VQ-VAE ESM3 ELBO OpenFold2 DeepSF
Closest papers by meaning
- ProteinBench: A Holistic Evaluation of Protein Foundation Models (ICLR 2025 · similarity 0.89)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.89)
- FoldToken: Learning Protein Language via Vector Quantization and Beyond (AAAI 2025 · similarity 0.89)
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.88)
- SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.88)
- Ensembits: an alphabet of protein conformational ensembles (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.