Research map / Vision-language and multimodal / Cross-modal learning
A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills morphological (microscopy) knowledge into transcriptomics encoders via a frozen-encoder CLIP-style adapter (Semi-Clipped) plus PEA, a batch-correction-based embedding augmentation, evaluated on known-relationship recall and interpretability across three OOD datasets.
Problem
Weakly paired microscopy-transcriptomics data is scarce, so transcriptomics representations cannot easily benefit from the richer predictive morphological features at inference time, which only uses transcriptomics.
Approach
Freeze pretrained unimodal encoders (Phenom-1 for images, scVI or scGPT for transcriptomics), train only a transcriptomics adapter with a CLIP loss against frozen image embeddings (Semi-Clipped), and augment student embeddings with randomized batch-correction transformations using random control-sample subsets (PEA).
Key results
- On HUVEC-KO, Semi-Clipped + PEA reaches 23.05 known-relationship recall vs 16.51 for the unimodal baseline (averaged over 15 seeds).
- PEA improves known-relationship recall for every distillation method tested (KD, SHAKE, VICReg, Semi-Clipped) on all three OOD datasets with p<0.05.
- Semi-Clipped + PEA reaches 19.63 recall on LINCS and 39.84 on SC-RPE1, the best known-relationship recall among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · HUVEC-KO | Known Relationships: 19.71 -> 23.05 | Helped |
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · LINCS | Known Relationships: 12.68 -> 19.63 | Helped |
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · SC-RPE1 | Known Relationships: 32.65 -> 39.84 | Helped |
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · HUVEC-KO | Tx Preservation: 52.78 -> 53.87 | Helped |
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · LINCS | Tx Preservation: 92.71 -> 93.15 | Helped |
| Add PEA augmentation to Semi-Clipped (ablation) | Semi-Clipped · SC-RPE1 | Tx Preservation: 37.54 -> 37.56 | No clear effect |
| Use fixed biological batch-correction augmentation (ablation) | Semi-Clipped · HUVEC-KO | Known Relationships: 19.71 -> 19.95 | Helped |
| Apply TVN correction at inference (ablation) | Semi-Clipped + fixed bio-augmentation · HUVEC-KO | Known Relationships: 19.95 -> 20.58 | Helped |
| Randomly drop batch-correction steps (ablation) | Semi-Clipped + inference on TVN · HUVEC-KO | Known Relationships: 20.58 -> 21.79 | Helped |
| Randomly sample a varying number of control samples (ablation) | Semi-Clipped + augmentation stochasticity · HUVEC-KO | Known Relationships: 21.79 -> 23.05 | Helped |
| Use PEA augmentation | Semi-Clipped · HUVEC-KO | Known Relationships: +17% | Helped |
| Use PEA augmentation | Semi-Clipped · LINCS | Known Relationships: +55% | Helped |
Surprising findings
- Label-free distillation (Semi-Clipped, VICReg) consistently outperforms label-based distillation (KD, SHAKE) in the reported comparisons.
- On LINCS and SC-RPE1, Semi-Clipped's transcriptomics preservation slightly trails the unimodal baseline, which the authors attribute to unseen cell types.
Limitations
- Random pairing within treatment groups may dilute representation quality when subtle intra-group differences exist.
- Limited large-scale paired data restricts broader applicability.
- Some PEA-augmented distillation methods show small drops in transcriptomics preservation (e.g., SHAKE and VICReg on SC-RPE1).
Experimental setup
Datasets: HUVEC-CMPD HUVEC-KO LINCS SC-RPE1
Benchmarks: HUVEC-KO LINCS SC-RPE1
Metrics: known biological relationship recall transcriptomic interpretability preservation (structural int
Baselines: CLIP SigCLIP VICReg DCCA KD SHAKE C2KD MWO MDWGAN-GP
Builds on: CLIP scVI scGPT Phenom-1 TVN
Closest papers by meaning
- Transcriptomics-guided Slide Representation Learning in Computational Pathology (CVPR 2024 · similarity 0.87)
- ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy (ICML 2025 · similarity 0.86)
- CellCLIP - Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (CVPR 2024 · similarity 0.86)
- Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images (ICLR 2025 · similarity 0.86)
- Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.