Research map / Probabilistic and scientific modeling / Molecules and proteins

Metalic: Meta-Learning In-Context with Protein Language Models

ICLR 2025 · Probabilistic and scientific modeling › Molecules and proteins · approach: Protein fitness and variant effect prediction · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Metalic, which meta-trains an in-context ProteinNPT-style axial-attention model over ProteinGym fitness tasks with a pairwise preference loss, then fine-tunes at test time on the support set without meta-gradients.

Problem

Protein fitness prediction in low-data or zero-data settings, where PLMs rely on strong assumptions (e.g., fitness correlates with sequence likelihood) and lack task-specific data.

Approach

Meta-learn in-context over a distribution of ProteinGym fitness tasks using ESM2-8M embeddings and axial attention with a pairwise ranking loss, then fine-tune on the target support set (with support sub-sampling to avoid memorization), skipping meta-gradients entirely.

Key results

• Zero-shot Spearman 0.484 (max) on held-out ProteinGym single-mutant tasks, above VESPA (0.464) and ESM1v-650M (0.437), with 18x fewer parameters.
• At 128 shots, Metalic reaches 0.552 ± 0.009 and Metalic-AuxIF 0.556 ± 0.005, comparable to ESM1v-650M (0.553) but below PoET (0.588).
• Without meta-training (NoMetaTrain) zero-shot Spearman drops to -0.046, showing meta-training drives zero-shot gains.

Measured effects reported in the paper

Surprising findings

• Metalic-AuxIF's multi-mutant performance improves when trained only on multi-mutant tasks, suggesting single-mutant data can discourage reliance on auxiliary inverse-folding predictions.
• Reptile performs poorly in isolation compared with in-context meta-learning, and combining Reptile with Metalic yields only marginal gains at 3x compute.

Limitations

• Does not achieve SOTA on multi-mutant tasks, attributed to limited multi-mutant training data.
• Hyperparameters were tuned relatively little, mostly on a single seed.
• Tasks with maximum protein length above 750 were excluded due to A100 memory limits.

Experimental setup

Datasets: ProteinGym

Benchmarks: ProteinGym

Metrics: Spearman correlation

Baselines: VESPA TranceptEVE-Medium ESM1v-650M Tranception-Medium ProGen2-Medium ESM2-650M MSA Transformer ESM-IF1 PoET ProteinNPT Reptile

Builds on: ProteinNPT ESM2 Reptile axial attention

Closest papers by meaning

• DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.89)
• Multi-Scale Representation Learning for Protein Fitness Prediction (NeurIPS 2024 · similarity 0.89)
• Steering Generative Models with Experimental Data for Protein Fitness Optimization (NeurIPS 2025 · similarity 0.88)
• From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.88)
• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.88)
• Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.87)