Research map / Vision-language and multimodal / Human and embodied modeling
STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STimage-1K4M, a dataset of 1,149 spatial transcriptomics slides split into 4.29M sub-tile histopathology images paired with 15k-30k dimensional gene expression, and fine-tunes CLIP/PLIP image encoders with gene-expression encoders as a demonstration.
Problem
Existing histopathology image-text datasets provide only slide-level text, lacking sub-tile annotations, and existing image-gene expression datasets are small and limited in scope.
Approach
Pairs sub-tile histopathology images from sequencing-based spatial transcriptomics (Spatial Transcriptomics, Visium, VisiumHD) with whole-transcriptome spot-level gene expression at large scale, and uses contrastive fine-tuning of CLIP and PLIP image encoders with gene expression encoders to show utility.
Key results
- STimage-1K4M contains 1,149 slides and 4,293,195 sub-tile image-gene expression pairs spanning 10 species and 50 tissue types.
- Fine-tuned CLIP and PLIP with HVG gene selection achieve higher mean macro F1 in linear probing than zero-shot CLIP and PLIP on DLPFC data.
- Fine-tuned image embeddings score better on cluster quality metrics and separate white matter from cortical layers better than zero-shot embeddings, though they remain below UNI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP on STimage-1K4M using HVG | CLIP · Human DLPFC | macro F1 | Helped |
| Fine-tune PLIP on STimage-1K4M using HVG | PLIP · Human DLPFC | macro F1 | Helped |
| Fine-tune image encoders using image–gene expression contrastive learning | CLIP and PLIP · Pathologist-annotated brain layers | cluster quality | Helped |
Surprising findings
- Fine-tuning on a single-tissue subset (other tissue types) yielded sub-optimal performance, attributed to batch effects across datasets.
Limitations
- UNI was not fine-tuned due to limited computational resources.
- Gene expression compressed to 128 dimensions and a 32-dimensional latent, possibly losing information.
- Data biased toward brain and breast cancer tissues; batch effects hinder multi-tissue training.
Experimental setup
Datasets: STimage-1K4M DLPFC spatial transcriptomics slides
Benchmarks: STimage-1K4M DLPFC linear probing
Metrics: macro F1 silhouette score Calinski-Harabasz index Davies-Bouldin index
Baselines: CLIP PLIP UNI
Builds on: CLIP PLIP
Closest papers by meaning
- M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images (AAAI 2025 · similarity 0.92)
- HEST-1k: A Dataset For Spatial Transcriptomics and Histology Image Analysis (NeurIPS 2024 · similarity 0.91)
- Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Images (ICLR 2025 · similarity 0.91)
- Preserving DEG Rankings for Gene Discovery in Histology-Based Spatial Gene Expression Prediction (NeurIPS 2026 · similarity 0.89)
- MEATRD: Multimodal Anomalous Tissue Region Detection Enhanced with Spatial Transcriptomics (AAAI 2025 · similarity 0.89)
- Scalable Generation of Spatial Transcriptomics from Histology Images via Whole-Slide Flow Matching (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.