Research map / Time series / Foundation models for time series
The Multimodal Universe: Enabling Large-Scale Machine Learning with 100 TB of Astronomical Scientific Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Multimodal Universe, a 100 TB community-curated, standardized multimodal dataset of astronomical images, spectra, time series and tabular data, with baseline benchmarks and a reproduced AstroCLIP model.
Problem
Astronomical survey data are experiment-specific, non-uniformly stored and not optimized for machine learning, and no large multimodal scientific dataset of comparable scale to web-scale text-image corpora exists for astrophysics.
Approach
A community-driven pipeline where domain scientists contribute download/format scripts that convert many survey archives into a common Hugging Face dataset schema per modality, plus cross-matching utilities to build multimodal subsets and baseline benchmarks.
Key results
- Pretraining on Galaxy Zoo gives ConvNext-nano 89.3% top-1 accuracy on Galaxy10 DECaLS vs 75.6% with no pretraining.
- Conv+Att on DESI spectra reaches R2 0.982 for redshift prediction, outperforming image-based ResNet18 (0.771).
- Zero-shot kNN on AstroCLIP embeddings gives R2 0.986 for redshift from DESI spectra.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Galaxy Zoo pretraining rather than no pretraining for ConvNext-nano | ConvNext-nano on Galaxy10 DECaLS · Galaxy10 DECaLS | Top-1 Accuracy: 75.6 ±1.8 % -> 89.3 ±0.1 % | Helped |
| Use ImageNet-12k pretraining rather than no pretraining for ConvNext-nano | ConvNext-nano on Galaxy10 DECaLS · Galaxy10 DECaLS | Top-1 Accuracy: 75.6 ±1.8 % -> 83.9 ±0.3 % | Helped |
| Use AstroCLIP image embeddings instead of DenseNet121 image predictions | Galaxy property prediction from Legacy Surveys images · Galaxy property prediction (Legacy Surveys images) | R2 ZHP: 0.774 -> 0.801 | Helped |
| Use AstroCLIP spectrum embeddings instead of Conv+Att spectrum predictions | Galaxy property prediction from DESI spectra · Galaxy property prediction (DESI spectrum) | R2 ZHP: 0.982 -> 0.986 | Helped |
Surprising findings
- Pretraining on ImageNet-12k (83.9%) underperforms Galaxy Zoo pretraining (89.3%) for galaxy morphology, and ImageNet pretraining gives a gain over none but not over domain data.
Limitations
- No textual captions or paired text corpora are included.
- Modalities are not guaranteed to be fully cross-matched across all sources.
- Baselines are simple supervised models rather than state-of-the-art multimodal foundation models.
Experimental setup
Datasets: Legacy Surveys DR10 HSC BTS JWST Gaia BP/RP SDSS-II DESI APOGEE GALAH Chandra VIPERS MaNGA
Benchmarks: Galaxy10 DECaLS PLAsTiCC YSE ZTF Bright Transient Survey PROVABGS galaxy property prediction
Metrics: top-1 accuracy R2 AUC RMSE
Baselines: EfficientNetB0 ResNet18 DenseNet121 ConvNext-nano Conv+Att MLP BTSbot Random forest Avocado Connect Later
Builds on: AstroCLIP CLIP Galaxy10 DECaLS PLAsTiCC
Closest papers by meaning
- CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learning (NeurIPS 2025 · similarity 0.85)
- Learning What's Real: Disentangling Signals and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics (NeurIPS 2026 · similarity 0.84)
- AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy (NeurIPS 2025 · similarity 0.83)
- A systematic evaluation of vision-language models for observational astronomical reasoning tasks (NeurIPS 2026 · similarity 0.83)
- MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens (NeurIPS 2024 · similarity 0.83)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.