Research map / Label-efficient and robust learning / Self-supervised and contrastive

Learning What's Real: Disentangling Signals and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a dual-encoder conditional flow-matching model that separates physics and instrument latents from overlapping multi-instrument galaxy images (HSC and Legacy), using counterfactual generation as the training objective.

Problem

Observations are confounded by instrument-specific artifacts, which hinder extracting physical signals and combining heterogeneous multi-instrument data; standard foundation models treat instruments as independent modalities without disentangling them.

Approach

Train two ResNet-18 encoders (physics encoder on the same source from another instrument; instrument encoder on other sources from the same instrument) whose latents condition a flow-matching decoder that reconstructs an anchor observation never fed to the encoders, so disentanglement emerges from the counterfactual generation objective rather than contrastive or hand-designed losses.

Key results

• Physics latents match AION-1 within a few percentage points on most physics targets (e.g., redshift R^2 0.801 vs 0.857 for AION-1-B) while instrument latents outperform AION-1 on instrument properties (galaxy depth R^2 0.682 vs 0.426).
• Outliers from the physics latent are free of instrument artifacts and show mergers, while AION-1 and instrument-latent outliers are dominated by saturation and noise patterns.
• Counterfactual HSC images from Legacy inputs yield ellipticity R^2 of 0.81 with a frozen HSC-trained ResNet, versus 0.82 on real HSC images.

Surprising findings

• Physics latents retain instrument-related information only near the random-baseline level, while a randomly initialized ResNet exceeds the trained instrument encoder on some physical properties.
• Cross-predict baseline outperforms the instrument latent on extinction-like quantities.

Limitations

• Requires overlapping observations across instruments, limiting applicability to shared sky regions.
• Generated counterfactuals reflect statistical expectation and cannot recover information absent from the input; mild posterior overconfidence (~15% variance underestimate).
• Evaluated only on galaxy imaging with two instruments; a simplified codebase is released but not the full pipeline.

Experimental setup

Datasets: Multimodal Universe cross-matched HSC and DESI Legacy Imaging Survey galaxy images PROVABGS catalogue HSC shape catalog

Benchmarks: Multi-instrument galaxy probe tasks (MLP R^2 on physics and instrument properties) Outlier detection on HSC galaxies Cross-instrument reconstruction MSE

Metrics: R^2 MSE log-likelihood (outlier score) power spectral density agreement pixel-wise Z-score calibration

Baselines: AION-1 AstroCLIP randomly initialized ResNet-18 cross-predict ResNet

Builds on: flow matching beta-VAE AION-1 UNet2DConditionModel ResNet

Closest papers by meaning

• When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.84)
• The Multimodal Universe: Enabling Large-Scale Machine Learning with 100 TB of Astronomical Scientific Data (NeurIPS 2024 · similarity 0.84)
• A systematic evaluation of vision-language models for observational astronomical reasoning tasks (NeurIPS 2026 · similarity 0.83)
• MultimodalStudio: A Heterogeneous Sensor Dataset and Framework for Neural Rendering across Multiple Imaging Modalities (CVPR 2025 · similarity 0.82)
• AstroCompress: A benchmark dataset for multi-purpose compression of astronomical data (ICLR 2025 · similarity 0.82)
• The Perception–Physics Paradox: Probing Scientific Alignment with TC-Bench (ICML 2026 · similarity 0.82)