Research map / Probabilistic and scientific modeling / Molecules and proteins
FoldToken: Learning Protein Language via Vector Quantization and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a discrete protein "foreign language" of FoldTokens jointly encoding residue types and backbone geometry via a new soft conditional vector quantizer (SoftCVQ), and uses it for a GPT-style sequence-structure co-generation model (FoldGPT) on backbone inpainting and antibody CDR design.
Problem
Protein structures are continuous 3D coordinates, which are hard to model with discrete sequence language models; existing VQ methods reconstruct sequences well but structures poorly, and there is no discrete protein language unifying sequence and structure for autoregressive generation.
Approach
Project residue types and backbone internal angles jointly into a discrete code space using SoftCVQ: soft attention queries the entire codebook (for reconstruction quality) while the codebook entries are binary vectors mapped through an MLP ConditionNet (for generation-friendly binary VQ-IDs), and a GPT-style model is trained over the resulting FoldToken sequences.
Key results
- SoftCVQ (6 MLP layers, 32-dim code, sphere normalization) reaches 0.9880 recovery and 0.7470 TMScore on CATH4.3 reconstruction, versus 0.4224 TMScore for vanilla VQ and 0.4385 for LFQ.
- FoldGPT achieves 96.2 sequence recovery vs 95.3 for ESM2 on backbone sequence inpainting, and angle-based structure TM 0.80 vs 0.74 for DiffSDS.
- On CDR-H3 antibody design, FoldGPT (antibody-only, C-alpha only) reports 23.37% AAR and 5.82 RMSD, outperforming the antibody-only LSTM and RefineGNN baselines but trailing antigen-aware ADesigner (37.37% AAR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Vanilla quantization with SoftCVQ | FoldTokenizer · CATH4.3 test set | TMScore: 0.4224 -> 0.7470 | Helped |
| replace LFQ with SoftCVQ | FoldTokenizer · CATH4.3 test set | TMScore: 0.4385 -> 0.7470 | Helped |
| use FoldGPT for sequence inpainting | sequence inpainting · CATH4.3 | sequence recovery: 95.3 -> 96.2 | Helped |
| use FoldGPT for angle-based structure inpainting | angle-based structure inpainting · CATH4.3 | TMScore: 0.74 -> 0.80 | Helped |
| use FoldGPT for CDR-H1 design | antibody-only design · CDR-H1 | AAR: 49.98$\pm$5.20\% -> 50.75$\pm$5.91\% | Helped |
| use FoldGPT for CDR-H2 design | antibody-only design · CDR-H2 | AAR: 37.06$\pm$3.09\% -> 39.71$\pm$3.02\% | Helped |
| use FoldGPT for CDR-H3 design | antibody-only design · CDR-H3 | AAR: 21.13$\pm$1.59\% -> 23.37$\pm$2.92\% | Helped |
| use 2 MLP layers, code dimension 16, and spherical normalization rather than no normalization (ablation) | SoftCVQ · CATH4.3 test set | TMScore: 0.5853 -> 0.6451 | Helped |
| use 2 MLP layers, code dimension 32, and spherical normalization rather than no normalization (ablation) | SoftCVQ · CATH4.3 test set | TMScore: 0.4285 -> 0.6959 | Helped |
| use 3 MLP layers, code dimension 32, and spherical normalization rather than no normalization (ablation) | SoftCVQ · CATH4.3 test set | TMScore: 0.5867 -> 0.6697 | Helped |
| increase MLP layers from 6 to 9 (ablation) | SoftCVQ · CATH4.3 test set | TMScore: 0.7470 -> 0.7015 | Hurt |
| use SoftVQ’s class-space language instead of SoftCVQ’s binary VQ-ID (ablation) | FoldGPT · backbone inpainting training | training convergence | Hurt |
Surprising findings
- SoftVQ with a 2^16 class space caused training crashes in generation, while the binary-ID SoftCVQ converged smoothly.
- Vanilla VQ and LFQ achieve high sequence recovery (>0.94) but structure TMScore below 0.5, showing sequence and structure reconstruction are very unequal.
Limitations
- FoldGPT uses angle-based structure representation that ignores antigen and other backbone atoms, limiting antibody design performance.
- Baselines were used via pretrained checkpoints and may be overstated because test data could overlap their training sets; FoldGPT was trained on less data.
- Generation is limited to backbone inpainting and antibody CDR design; predictive tasks were not evaluated.
Experimental setup
Datasets: cAF2DB CATH4.3 SAbDab
Benchmarks: CATH4.3 test set CATH4.3 inpainting test set (117 proteins) SAbDab CDR-H1/H2/H3
Metrics: sequence recovery TMScore recovery success rate reconstruction success rate local L_r L_alpha L_beta AAR RMSD
Baselines: VVQ LFQ FoldingDiff DiffSDS ESM2 EvoDiff ProtDiff Chroma RFDiffusion DiffAB MEAN ADesigner
Builds on: VQ-VAE vanilla vector quantization lookup-free quantization GLM FoldingDiff
Closest papers by meaning
- Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.89)
- ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (NeurIPS 2024 · similarity 0.88)
- DPLM-2: A Multimodal Diffusion Protein Language Model (ICLR 2025 · similarity 0.88)
- ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Search (NeurIPS 2025 · similarity 0.88)
- Protein Design with Dynamic Protein Vocabulary (NeurIPS 2025 · similarity 0.87)
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.