Research map / Vision-language and multimodal / Cross-modal learning
CellCLIP - Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CellCLIP, a text-guided cross-modal contrastive framework for Cell Painting that pools per-channel DINOv2 profiles with a CrossChannelFormer, aligns them with BERT-encoded perturbation prompts, and uses a continuously weighted contrastive loss.
Problem
Aligning Cell Painting morphological profiles with perturbation labels across heterogeneous perturbation classes (compounds, CRISPR, ORF) is hard due to multi-channel semantics, many-to-one image-perturbation structure causing false negatives, and high compute cost of channel-aware encoders.
Approach
Treat each Cell Painting channel as a grayscale image embedded by a frozen natural-image foundation model (DINOv2), pool profiles per perturbation with gated attention (ABMIL), reason across channels with a small transformer using channel embeddings and a CLS token, encode perturbations as natural-language prompts with BERT, and train with a continuously weighted (CWCL) loss on profile-to-perturba
Key results
- On unseen-compound retrieval, CellCLIP reaches 1.18% perturb-to-profile R@1 and 7.37% R@10, versus 0.56% R@1 and 4.30% R@10 for the best baseline MolPhenix* (SigCLIP).
- CellCLIP achieves replicate detection mAP of 0.663 on CP-JUMP1 versus 0.575 for CLOOME, and cross-class sister matching mAP of 0.043.
- CrossChannelFormer with pooling trains in 1.81 hours versus 49.9 hours for the CLOOME ResNet-101 baseline, with reported 6.7x speedup from pooling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CellCLIP instead of MolPhenix* (SigCLIP) | unseen-molecule cross-modal retrieval · unseen molecules, perturbation-to-profile | Recall@10 (%): 4.30 \( \pm 0.16 \) -> 7.37 \( \pm \) 0.20 | Helped |
| use CellCLIP instead of MolPhenix* (SigCLIP) | unseen-molecule cross-modal retrieval · unseen molecules, profile-to-perturbation | Recall@10 (%): 4.34 \( \pm \) 0.24 -> 7.39 \( \pm \) 0.23 | Helped |
| replace Morgan fingerprints plus MLP with text prompts and BERT (ablation) | CLOOME-style ResNet-101 retrieval model · unseen molecules, perturbation-to-profile | Recall@10 (%): 2.46 \( \pm \) 0.56 -> 4.50 \( \pm\) 0.29 | Helped |
| remove SMILES string from prompt (ablation) | CellCLIP · unseen molecules, perturbation-to-profile | Recall@10 (%): 7.37 \pm 0.20 -> 1.26 \pm 0.14 | Hurt |
| remove cell type information from prompt (ablation) | CellCLIP · unseen molecules, perturbation-to-profile | Recall@10 (%): 7.37 \pm 0.20 -> 6.59 \pm 0.33 | Hurt |
| remove drug name from prompt (ablation) | CellCLIP · unseen molecules, perturbation-to-profile | Recall@10 (%): 7.37 \pm 0.20 -> 7.37 \pm 0.29 | No clear effect |
| add DrugBank description to prompt (ablation) | CellCLIP · unseen molecules, perturbation-to-profile | Recall@10 (%): 7.37 \pm 0.20 -> 6.94 \pm 0.37 | Hurt |
| add CrossChannelFormer channel-aware image encoding (ablation) | CLOOME with individual image supervision · unseen molecules, perturbation-to-profile | Recall@10 (%): 4.50 \pm 0.29 -> 5.74 \pm 0.36 | Helped |
| pool image profiles per perturbation with attention pooling (ablation) | CrossChannelFormer · unseen molecules, perturbation-to-profile | Recall@10 (%): 5.74 \pm 0.36 -> 7.37 \( \pm 0.20 | Helped |
| replace attention pooling with median pooling (ablation) | CellCLIP · unseen molecules, perturbation-to-profile | Recall@10 (%): 7.37 \( \pm 0.20 -> 6.26 \( \pm 0.30 | Hurt |
| replace CWCL with standard CLIP loss (ablation) | CellCLIP · RxRx3-core gene relationship recovery | Recall: .714 -> .673; .778 -> .751; .427 -> .416; .388 -> .370; .618 -> .595 | Hurt |
| replace CWCL with SigCLIP loss (ablation) | CellCLIP · RxRx3-core gene relationship recovery | Recall: .714 -> .741; .778 -> .800; .427 -> .466; .388 -> .454; .618 -> .646 | Helped |
Surprising findings
- Removing the SMILES string from the prompt collapses retrieval performance (R@10 falls from 7.37% to 1.26%), while removing drug names has little effect.
- OpenPhenom-S/16, a Cell Painting-pretrained model, did not outperform DINO models pretrained on natural images within CellCLIP.
Limitations
- Evaluation limited to datasets from previous works with no wet-lab validation.
- Many Cell Painting datasets and models remain proprietary, limiting reproducibility.
- Retrieval performance is low in absolute terms (R@1 about 1%), and cross-class sister matching remains weak (mAP 0.043).
Experimental setup
Datasets: RxRx3-core CP-JUMP1 curated RxRx-style small-molecule Cell Painting dataset (284,034 images) Reactome HuMAP SIGNOR StringDB CORUM DrugBank
Benchmarks: Cell Painting cross-modal retrieval (unseen compounds) CP-JUMP1 replicate detection CP-JUMP1 sister perturbation matching RxRx3-core gene-gene relationship recovery
Metrics: Recall@1 Recall@5 Recall@10 mean average precision recall at top/bottom percentile thresholds
Baselines: CLOOME MolPhenix* OpenPhenom-S/16 ViT-L/16 ChannelViT CA-MAE
Builds on: CLIP CLOOME MolPhenix CWCL ABMIL
Closest papers by meaning
- MorphoHELM: A Comprehensive Benchmark for Evaluating Representations for Microscopy-Based Morphology Assays (NeurIPS 2026 · similarity 0.88)
- ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy (ICML 2025 · similarity 0.87)
- A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features (ICML 2025 · similarity 0.86)
- How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval (NeurIPS 2024 · similarity 0.86)
- Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding (NeurIPS 2024 · similarity 0.86)
- CellFlux: Simulating Cellular Morphology Changes via Flow Matching (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.