Research map / Vision-language and multimodal / Cross-modal learning

ES-Merging: Biological MLLM Merging via Embedding Space Signals

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ES-Merging, which merges modality-specialized biological MLLMs (molecule, protein, cell) by estimating layer-wise (SWD-based) and element-wise (gradient-based) merging coefficients from embedding-space signals on a shared probe input.

Problem

Merging single-modality biological MLLMs into one unified cross-modal model; existing merging methods rely on input-agnostic parameter-space heuristics that poorly capture modality specialization.

Approach

Derive merging coefficients from input-conditioned embedding-space discrepancies between the base LLM and each specialized MLLM on a probe input: coarse layer-wise coefficients from changes in sliced Wasserstein distance over mean-pooled embeddings, and fine element-wise coefficients from gradients of Frobenius embedding distance with respect to LoRA parameters, combined multiplicatively.

Key results

• On molecule-protein interaction, ES-Merging reaches 65.7% average accuracy vs 64.5% for Avg. Merging and 57.5% for the base LLaMA-3.1-8B-Instruct.
• On single-modal knowledge preservation, ES-Merging averages 75.21 vs 63.72 for Avg. Merging and 79.59 for the specialized MLLMs.
• Total FLOPs for determining merged LoRA parameters is 149,807 vs 891,117 for Avg. Merging + FT (5.95x reduction).

Surprising findings

• Parameter-signal merging methods such as AdaMerging and UQ-Merge drop sharply on the molecule-cell DrugComb task (e.g., AdaMerging 16.0 accuracy), indicating instability across datasets.
• ES-Merging marginally surpasses specialized MLLMs on molecule and protein single-modal tasks.

Limitations

• Only three biological specialists and a single shared base LLM were merged, limiting generality of the findings.
• Cell modality performance remains below the cell-specialized MLLM after merging.
• Several baselines (e.g., EMR-Merging, AdaMerging) show unstable performance across datasets, and results rely on few-shot in-context prompts.

Experimental setup

Datasets: BindingDB BioSNAP Human DrugComb GDSC2 CYP1A2 CYP2C19 CYP2C9 CYP2D6 CYP3A4 TDC CYP substrate PAMPA

Benchmarks: BindingDB BioSNAP Human DrugComb GDSC2 CYP inhibition CYP substrate PAMPA Immuno-Virus PBMC MUSIC-AVQA

Metrics: accuracy macro-F1

Baselines: Average Merging TIES-Merging EMR-Merging AdaMerging PCB-Merging Consensus Merging UQ-Merge RobustMerge LS-Merge Avg. Merging + FT Mol-LLaMA Prot2Text-V2

Builds on: TIES-Merging AdaMerging LoRA

Closest papers by meaning

• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.86)
• Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.85)
• Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine (ICLR 2025 · similarity 0.85)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.85)
• Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.85)
• Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging (NeurIPS 2026 · similarity 0.84)