Research map / Time series / Foundation models for time series
AstroCompress: A benchmark dataset for multi-purpose compression of astronomical data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AstroCompress, a 16-bit astronomical imaging dataset corpus (~320 GB, five datasets) and benchmarks three neural and four classical lossless codecs on it, finding VDM and JPEG-XL (max) strongest.
Problem
Lossless compression of raw 16-bit astronomical imaging data is bottlenecked by limited telescope data transmission, yet neural lossless compression has not been systematically studied on astronomy data and no ML-friendly raw corpus existed.
Approach
Curate a large, non-overlapping corpus of raw uint16 astronomical images (2D, 3D time-series, and 4D wavelength-time cubes) from space and ground telescopes, and benchmark neural and classical lossless codecs on it with minimal 16-bit adaptations of neural models.
Key results
- VDM achieves the highest estimated compression ratios on most single-frame datasets (e.g., 3.64 on LCO, 3.33 on Hubble), with the paper noting these are likelihood-based estimates.
- JPEG-XL (max effort) is the best non-neural method across datasets, e.g., 2.98 on LCO and 3.26 on Hubble, and is the strongest codec on the correlated multi-frame datasets (3.35 on JWST-2D-Res, 3.49 on SDSS-3D-lambda).
- Neural codecs IDF and PixelCNN++ are competitive on single-frame data but underperform non-neural codecs on spectrally/temporally correlated data, e.g., IDF 3.05 vs JPEG-XL 3.49 on SDSS-3D-lambda.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VDM likelihood estimates instead of the best non-neural JPEG-XL setting | Lossless compression on LCO · LCO | compression ratio: 2.98 -> 3.64 | Helped |
| Use VDM likelihood estimates instead of the best non-neural JPEG-XL setting | Lossless compression on Keck · Keck | compression ratio: 2.01 -> 2.11 | Helped |
| Use VDM likelihood estimates instead of the best non-neural JPEG-XL setting | Lossless compression on Hubble · Hubble | compression ratio: 3.26 -> 3.33 | Helped |
| Use JPEG-XL (max) instead of IDF | Lossless compression on JWST-2D · JWST-2D | compression ratio: 1.44 -> 1.38 | Hurt |
| Use JPEG-XL (max) instead of PixelCNN++ | Lossless compression on SDSS-2D · SDSS-2D | compression ratio: 3.35 -> 3.38 | Helped |
| Use JPEG-XL (max) instead of IDF | Lossless compression on JWST-2D-Res · JWST-2D-Res | compression ratio: 3.14 -> 3.35 | Helped |
| Use JPEG-XL (max) instead of IDF | Lossless compression on SDSS-3Dλ · SDSS-3Dλ | compression ratio: 3.05 -> 3.49 | Helped |
| Use JPEG-XL (max) instead of IDF | Lossless compression on SDSS-3DT · SDSS-3DT | compression ratio: 3.03 -> 3.48 | Helped |
| Train IDF on all datasets combined rather than on one individual dataset | IDF · SDSS-2D | compression ratio: 2.91 -> 3.18 | Helped |
| Train IDF on Keck rather than SDSS-2D | IDF · SDSS-2D | compression ratio: 2.91 -> 3.02 | Helped |
| Represent 16-bit pixels as two 8-bit channels (MSB and LSB) instead of direct 16-bit input (ablation) | Neural compression methods · AstroCompress datasets | compression ratio | Mixed |
| Use cross-frame correlations in residual coding | Lossless compression on JWST · JWST-2D-Res | compression ratio | Helped |
Surprising findings
- Non-neural codecs gain more from cross-wavelength and cross-time correlations than neural codecs do, which the authors attribute to neural methods being designed for natural images.
- Training IDF on the Keck dataset generalized better to SDSS-2D than training on SDSS-2D itself.
Limitations
- Neural compression ratios are estimated from model likelihoods without entropy coding; the authors note the implementation overhead is small but actual codecs were not built.
- Neural methods operate on 32x32 patches, which likely limits achievable compression ratio versus full-image classical baselines.
- The corpus lacks spectroscopic, radio, and floating-point data, and neural methods for 3D cubes are underexplored.
Experimental setup
Datasets: GBI-16-2D (Keck) SBI-16-2D (Hubble) SBI-16-3D (JWST) GBI-16-4D (SDSS) GBI-16-2D-Legacy (LCO)
Benchmarks: AstroCompress LCO AstroCompress Keck AstroCompress Hubble AstroCompress JWST-2D AstroCompress SDSS-2D AstroCompress JWST-2D-Res AstroCompress SDSS-3D-lambda AstroCompress SDSS-3DT
Metrics: compression ratio encoding runtime (sec/image)
Baselines: JPEG-XL JPEG-LS JPEG-2000 RICE IDF L3C PixelCNN++
Builds on: Integer Discrete Flows L3C PixelCNN++ Variational Diffusion Models JPEG-XL
Closest papers by meaning
- COSMIC: Compress Satellite Image Efficiently via Diffusion Compensation (NeurIPS 2024 · similarity 0.87)
- MeLLoC: Lossless Compression with High-order Mechanism Learning (NeurIPS 2024 · similarity 0.83)
- Probabilistic Data-Driven Modelling of Astrophysical Transients: The Neural Process Family for Ultrafast and Class-Agnostic Light Curve Reconstruction (NeurIPS 2026 · similarity 0.83)
- Learning What's Real: Disentangling Signals and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics (NeurIPS 2026 · similarity 0.82)
- The Multimodal Universe: Enabling Large-Scale Machine Learning with 100 TB of Astronomical Scientific Data (NeurIPS 2024 · similarity 0.82)
- UniPCGC: Towards Practical Point Cloud Geometry Compression via an Efficient Unified Approach (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.