Research map / Vision-language and multimodal / Cross-modal learning
Propensity Score Alignment of Unpaired Multimodal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes aligning unpaired multimodal samples by estimating a propensity score per modality from perturbation labels and matching within each perturbation via optimal transport (or shared nearest neighbours) on that score.
Problem
Aligning unpaired samples across modalities (e.g., single-cell RNA-seq and protein, or microscopy and expression) when each sample is measured in only one modality, so no ground-truth pairs exist; only experimental perturbation labels are observed.
Approach
Treats modalities as potential outcomes of a shared latent state perturbed by treatments; the propensity score p(t|x) trained as a classifier per modality is shown to be identical across modalities and used as a cross-modal distance for entropic OT or SNN matching within each perturbation.
Key results
- On synthetic images, PS+OT achieved median MSE 0.0316 and trace 18.329e-3, versus 0.0324 / 7.733e-3 for VAE+OT and 0.0709 for random.
- On CITE-seq, PS+OT reached FOSCTTM 0.3049 versus 0.3953 for VAE+OT, and downstream R^2 of 0.233 with unbiased loss versus 0.114 for VAE+OT.
- On PerturbSeq/single-cell images, PS+OT lowered in-distribution KL to 50.967 from 55.483 (VAE+OT) and 58.806 (random).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use propensity-score distances with OT instead of VAE embeddings with OT | Unpaired multimodal matching · Synthetic images | Trace: 7.733 -> 18.329 | Helped |
| Use propensity-score distances with OT instead of VAE embeddings with OT | Unpaired multimodal matching · CITE-seq | FOSCTTM: 0.3953 -> 0.3049 | Helped |
| Use propensity-score distances with OT instead of VAE embeddings with OT | Unpaired multimodal matching · CITE-seq | Trace: 0.0814 -> 0.1163 | Helped |
| Use propensity-score OT matching instead of VAE OT matching | Cross-modal prediction · CITE-seq | R², unbiased loss: 0.114 -> 0.233 | Helped |
| Train with propensity-score OT pseudosamples instead of true pairs | CITE-seq prediction model · CITE-seq | R², unbiased loss: 0.224 -> 0.233 | Mixed |
| Use propensity-score OT matching instead of VAE OT matching | Cross-modal prediction · PerturbSeq/single-cell images, out of distribution | KL: 47.910 -> 43.554 | Helped |
| Use propensity-score OT matching instead of VAE OT matching | Cross-modal prediction · PerturbSeq/single-cell images, in distribution | Wasserstein-1: 4.339 -> 4.199 | Helped |
| Use propensity-score OT matching instead of VAE OT matching | Cross-modal prediction · PerturbSeq/single-cell images, out of distribution | L2 norm: 7.444 -> 7.219 | Helped |
| Use unbiased projected loss instead of MSE loss (ablation) | PS+OT CITE-seq prediction model · CITE-seq | R²: 0.217 -> 0.233 | Helped |
Surprising findings
- PS+OT downstream R^2 (0.233) with unbiased loss was slightly above training on true pairs with MSE loss (0.224), though confidence intervals overlap.
- Reconstruction-based VAE representations gave worse matching than a classifier-only objective with identical architectures; validation cross-entropy of the propensity classifier tracked matching quality while VAE validation loss did not.
Limitations
- Relies on assumptions (A1) and (A2) that are strong; matching is impossible without shared information.
- Requires an experimental perturbation label that differs across units, and the propensity score may be non-injective when the number of perturbations is small relative to latent dimension.
- PerturbSeq/image dataset is private and evaluated only with distributional metrics without ground truth.
Experimental setup
Datasets: synthetic interventional images (PyGame rendered) NeurIPS 2021 Multimodal single-cell CITE-seq (GEO GSE194122) PerturbSeq HUVEC (in-house, 200 genes, 24 perturbations) HUVEC single-cell painting images (in-house)
Benchmarks: synthetic interventional images NeurIPS 2021 CITE-seq PerturbSeq/single-cell image cross-modal prediction
Metrics: trace latent MSE FOSCTTM R^2 KL divergence Wasserstein-1 L2 norm of mean difference
Baselines: multimodal VAE + OT multimodal VAE + SNN SCOT scGLUE + SNN scGLUE + OT random matching true pairs
Builds on: Sinkhorn entropic optimal transport multimodal VAE with prior information (Lopez et al.) scVI SCOT
Closest papers by meaning
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.88)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.87)
- Causal Representation Learning from Multimodal Biomedical Observations (ICLR 2025 · similarity 0.87)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.86)
- When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.