Research map / Label-efficient and robust learning / Recognition and long-tail
Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vabs-Net, a bilevel (residue and atom) sparse-attention protein structure pre-trained model, with a span mask strategy (SMPC) that removes side-chain atoms in masked spans to prevent residue-level information leakage.
Problem
Structure-based protein pre-training mostly models only residue-level (C-alpha) geometry, and naively adding atom-level information causes information leakage that makes residue-level pre-training tasks trivial.
Approach
Jointly model residues and atoms in a two-track sparse attention network with vector-aware edge encoding, and mask consecutive residue spans by keeping only C-alpha atoms (span mask on protein chains) so residue-type and torsion prediction cannot leak through side-chain atoms; atom-level pre-training uses Gaussian noise plus position prediction.
Key results
- Vabs-Net with pre-training reaches 0.531 BP, 0.695 MF, 0.579 CC and 0.900 EC F_max on GO/EC prediction, above GearNet-ESM (0.890 EC) and SiamDiff-ESM (0.897 EC).
- On small-molecule binding site prediction (COACH420), Vabs-Net with pre-training achieves 64.1 IoU versus 41.3 for GearNet and 40.3 for Siamdiff.
- On Equibind molecular docking, adding Vabs-Net features lowers ligand RMSD from 8.91 (Equibind) to 7.23 and centroid RMSD from 6.02 to 4.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SMPC span masking during pre-training (ablation) | Vabs-Net · EC | Fmax: 0.876 -> 0.900 | Helped |
| Add SMPC span masking during pre-training (ablation) | Vabs-Net · COACH420 small-molecule binding site | IoU (%): 63.0 -> 64.1 | Helped |
| Add atom-level encoding (ablation) | Vabs-Net · COACH420 small-molecule binding site | IoU (%): 56.3 -> 64.1 | Helped |
| Add residue-level encoding (ablation) | Vabs-Net · EC | Fmax: 0.886 -> 0.900 | Helped |
| Add vector edge encoder, retaining SMPC (ablation) | Vabs-Net · COACH420 small-molecule binding site | IoU (%): 60.9 -> 64.1 | Helped |
| Add ESM features, retaining SMPC (ablation) | Vabs-Net · EC | Fmax: 0.867 -> 0.900 | Helped |
| Increase KNN to 90 while removing SMPC (ablation) | Vabs-Net · COACH420 small-molecule binding site | IoU (%): 63.0 -> 62.7 | Hurt |
| Add pre-training (ablation) | Vabs-Net · EC | Fmax: 0.826 -> 0.900 | Helped |
| Use sparse attention instead of AttMLP (ablation) | Vabs-Net · COACH420 small-molecule binding site | IoU (%): 62.3 -> 64.1 | Helped |
| Use consecutive-span masking instead of random masking (ablation) | Vabs-Net · EC | Fmax: 0.884 -> 0.900 | Helped |
| Use Vabs-Net features instead of Siamdiff features in Equibind | Equibind · Molecular docking (PDBBind) | Ligand RMSD: 7.75 -> 7.23 | Helped |
| Add Vabs-Net pre-training | Vabs-Net · DNA-129 test | AUC: 0.912 -> 0.940 | Helped |
Surprising findings
- Naively combining residue and atom inputs in pre-training fails because atom context leaks residue identity, making residue-level tasks trivial.
- Masking only side-chain atoms (keeping backbone) or randomly masking non-consecutive residues does not fully prevent information leakage.
Limitations
- Pre-training dataset (about 130K structures) is much smaller than sequence pre-training corpora and the AlphaFold database.
- Pre-training is limited to single-chain proteins, ignoring inter-chain interactions.
- Molecular docking uses a modified Equibind with a single attention layer for atom-to-residue aggregation.
Experimental setup
Datasets: Protein Data Bank AlphaFold DB PDBBind CASF-2016 MOAD DNA-573 DNA-129 RNA-495 RNA-117 EC/GO benchmark COACH420
Benchmarks: EC GO COACH420 B277 DT198 ASTEX85 CHEN251 DNA-129 RNA-117 PDBBind
Metrics: F_max IoU AUC LIGAND RMSD CENTROID RMSD
Baselines: GearNet Siamdiff ESM2_150M GearNet-ESM SiamDiff-ESM PromptProtein FPocket P2Rank DNABind GraphBind Equibind
Builds on: GearNet Siamdiff Graphormer-3D PiFold ESM
Closest papers by meaning
- Atom-level Protein Representation Learning Improves Protein Structure Prediction (NeurIPS 2026 · similarity 0.89)
- AtomSurf: Surface Representation for Learning on Protein Structures (ICLR 2025 · similarity 0.89)
- MORE: Molecule Pretraining with Multi-Level Pretext Task (AAAI 2025 · similarity 0.88)
- Evaluating Representation Learning on the Protein Structure Universe (ICLR 2024 · similarity 0.87)
- Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling (ICML 2025 · similarity 0.87)
- Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.