Research map / Vision-language and multimodal / Human and embodied modeling
Elucidating the Design Space of Multimodal Protein Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies design choices for token-based multimodal protein language models (DPLM-2 family), adding bit-wise supervision, flow-matching hybrid sampling, geometry-aware modules, representation alignment to ESMFold, and multimer data, yielding a 650M model with much better folding RMSD.
Problem
Token-based multimodal protein language models lose fine-grained structural detail when 3D structures are quantized into discrete tokens, and index-based token prediction is inaccurate, which limits folding accuracy and structure generation diversity.
Approach
Improve DPLM-2 along several axes: bit-wise binary supervision instead of index classification, a residual diffusion module (ResDiff) to recover quantization residuals, a hybrid flow-matching sampler that uses the encoder-PLM-decoder as a denoiser, geometry-aware pair/triangle modules, REPA alignment to ESMFold trunk representations, and including multimer data.
Key results
- Folding RMSD on PDB date split drops from 5.52 (DPLM-2 650M) to 2.37 with bit-based modeling, FM, geometric modules and folding SFT, outperforming the 3B DPLM-2 baseline and approaching ESMFold (2.84).
- Bit-based supervision raises bit-level structure token accuracy on PDB date split from 0.7932 to 0.8648 and index accuracy from 0.1188 to 0.2641.
- REPA plus geometric modules on GeoDPLM improves CAMEO 2022 RMSD from 7.703 (DPLM-2) to 7.058 and TMscore to 0.8217.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bit-based rather than index-based structure modeling | DPLM-2 · PDB date split | RMSD: 5.3071 -> 3.2213 | Helped |
| add ResDiff (ablation) | DPLM-2 · CAMEO 2022 | RMSD: 7.7025 -> 7.2881 | Helped |
| add flow matching to bit-based DPLM-2 (ablation) | DPLM-2 Bit-based · PDB date split | RMSD: 3.221 -> 2.870 | Helped |
| add folding SFT to Bit + FM + ResDiff (ablation) | DPLM-2 Bit-based + FM + ResDiff · PDB date split | RMSD: 2.788 -> 2.370 | Helped |
| add pair bias and pair representation transition layers | DPLM-2 · PDB date split | RMSD: 5.521 -> 4.823 | Helped |
| add a structure representation transition layer (ablation) | GeoDPLM (Base) · PDB date split | RMSD: 4.823 -> 3.883 | Helped |
| add triangle update to GeoDPLM (ST) (ablation) | GeoDPLM · PDB date split | RMSD: 3.883 -> 4.837 | Hurt |
| add REPA representation alignment (ablation) | DPLM-2 · PDB date split | RMSD: 5.521 -> 4.919 | Helped |
| add REPA representation alignment (ablation) | GeoDPLM · PDB date split | RMSD: 4.823 -> 4.340 | Helped |
| align pair representations rather than no representations (ablation) | GeoDPLM, 150M · CAMEO 2022 | RMSD: 9.410 -> 9.467 | Hurt |
| increase monomer training data from 200K to 1.2M additional AFDB_Rep | Structure tokenizer · PDB-Multimer | RMSD: 9.973 -> 6.873 | Helped |
| add a position offset (ablation) | DPLM-2 tokenizer · PDB-Multimer reconstruction | RMSD: 6.873 -> 5.886 | Helped |
Surprising findings
- Triangle update and triangle attention operations give little folding benefit while slowing training by 3.9x and 6.8x.
- Combining bit-based modeling with REPA or flow matching does not give further gains, indicating non-orthogonal effects.
Limitations
- Relies on discrete residue-level tokens, so information loss and lack of atomic detail remain.
- Experiments limited to models up to 3B; scalability of the design choices is uncertain.
- Multimer training data is small (about 3.5K samples) and sparse.
Experimental setup
Datasets: PDB SwissProt AFDB_Rep PDB-Multimer
Benchmarks: CAMEO 2022 PDB date split PDB-Multimer DeepLoc Subcellular HumanPPI
Metrics: RMSD TMscore structure token accuracy diversity (FoldSeek clusters) AAR
Baselines: ESMFold MultiFlow ESM3 DPLM-2 (3B) SaProt
Builds on: DPLM-2 DPLM REPA AlphaFlow PairFormer
Closest papers by meaning
- DPLM-2: A Multimodal Diffusion Protein Language Model (ICLR 2025 · similarity 0.91)
- SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.90)
- Protein Design with Dynamic Protein Vocabulary (NeurIPS 2025 · similarity 0.89)
- Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.88)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.88)
- Steering Protein Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.