Research map / Continual learning / Continual fine-tuning of LLMs
Retrieval-Augmented Language Model for Knowledge-aware Protein Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kara, a retrieval-augmented protein language model that integrates a protein knowledge graph (ProteinKG25) into ProtBert via contextualized virtual tokens, a learned knowledge retriever for new proteins, and structure-based regularization applied in both pre-training and fine-tuning.
Problem
Protein language models lack factual/gene-description knowledge, and existing PKG-enhanced models use knowledge only as pre-training objectives, leading to limited knowledge exploitation, catastrophic forgetting, and no task-oriented knowledge integration for proteins outside the graph.
Approach
Align downstream (out-of-graph) proteins with the PKG via a retriever that predicts linked GO entities, then inject knowledge and two-hop structure as contextualized virtual tokens and use a shared structure-based margin regularizer across pre-training and fine-tuning.
Key results
- Kara improves over KeAP by 11.6% on long-range contact prediction and 10.3% on homology detection (0.32 vs 0.29).
- Kara reaches 0.83 stability Spearman vs 0.80 for KeAP and ESM-2, and 80.01 F1 on SHS27K PPI.
- Averaged over 6 tasks, Kara reports about 5.1% average improvement over existing knowledge-enhanced models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contextualized virtual tokens (ablation) | Kara · Protein-protein binding affinity (SKEMPI) | MSE: 0.55 -> 0.50 | Helped |
| Add structure-based regularization (ablation) | Kara · Protein homology detection | Accuracy: 0.30 -> 0.32 | Helped |
| Add structure-based regularization (ablation) | Kara · Protein stability prediction | Score: 0.80 -> 0.83 | Helped |
| Use the knowledge retriever instead of protein sequence similarity retrieval (ablation) | Kara · Protein homology detection | Accuracy: 0.29 -> 0.32 | Helped |
| Include neighboring structure information in candidate GO embeddings (ablation) | Knowledge retriever · ProteinKG25 retrieval | P@1: 0.681 -> 0.821 | Helped |
| Add cross-modal matching loss (ablation) | Knowledge retriever · ProteinKG25 retrieval | P@1: 0.733 -> 0.821 | Helped |
| Use relation-GO combinations to restrict candidates (ablation) | Knowledge retriever · ProteinKG25 retrieval | P@1: 0.649 -> 0.821 | Helped |
| Increase the number of retrieved knowledge items K from 0 to 1 (ablation) | Kara · Homology detection | Accuracy | Helped |
| Increase the number of retrieved knowledge items beyond 1 (ablation) | Kara · Downstream tasks | Performance | Mixed |
| Use incomplete ProteinKG25 rather than full PKG (ablation) | Kara · Contact prediction, 6≤seq≤12 | Contact: 0.546 -> 0.540 | Hurt |
| Change Kara backbone from ProtBert to ProteinBert (ablation) | Kara · Contact prediction, 6≤seq≤12 | Contact: 0.553 -> 0.556 | Helped |
| Change Kara backbone from ProtBert to ESM-1b (ablation) | Kara · Protein homology detection | Accuracy: 0.323 -> 0.327 | Helped |
Surprising findings
- Knowledge-enhanced models OntoProtein and KeAP fail to beat traditional protein LMs on homology detection and stability.
- A protein-similarity-based retriever does not outperform the learned knowledge retriever, since similar sequences can have different functions.
Limitations
- Structure modeling of 3D protein structure is absent, leading to weaker performance on long sequences (error analysis).
- Binding affinity and BP semantic similarity do not beat ESM-1b.
- Requires a pre-built protein knowledge graph and retrieval of GO entities; performance depends on retriever accuracy (P@1 0.821).
Experimental setup
Datasets: ProteinKG25 SHS27K SHS148K STRING SKEMPI TAPE homology/contact benchmarks stability dataset (Rocklin et al.)
Benchmarks: TAPE contact prediction SHS27K SHS148K STRING TAPE homology detection stability prediction SKEMPI binding affinity semantic similarity inference
Metrics: P@L F1 accuracy Spearman correlation MSE Precision@n
Baselines: ProtBert ESM-1b ESM-2 OntoProtein KeAP MSA Transformer LSTM ResNet Transformer PIPR DPPI DNN-PPI
Builds on: OntoProtein KeAP TransE ProtBert
Closest papers by meaning
- Knowledge-aware Reinforced Language Models for Protein Directed Evolution (ICML 2024 · similarity 0.87)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.86)
- Understanding protein function with a multimodal retrieval-augmented foundation model (NeurIPS 2025 · similarity 0.86)
- Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (ICML 2024 · similarity 0.86)
- Concept Bottleneck Language Models For Protein Design (ICLR 2025 · similarity 0.85)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.