Research map / Vision-language and multimodal / Human and embodied modeling
Galaxy Walker: Geometry-aware VLMs For Galaxy-scale Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Galaxy Walker is a geometry-aware VLM that adds random-walk geometry prompts over Euclidean, spherical and hyperbolic galaxy graphs and a mixture-of-experts geometry adapter on Qwen2-VL-2B, evaluated on galaxy property estimation and morphology classification.
Problem
General VLMs and domain models struggle to represent astronomical data whose geometry is non-Euclidean (spherical, hyperbolic), leading to poor galaxy property regression and morphology classification.
Approach
Builds multi-geometry galaxy graphs (Euclidean, spherical, hyperbolic) from sky coordinates to produce geometry tokens, and inserts a heterogeneous MoE adapter with Euclidean, spherical and hyperbolic FFN experts into a pretrained VLM, trained with a language-modeling loss plus a regression loss from a numeric head.
Key results
- Galaxy Walker reaches R2 0.91 on M* and 0.84 on sSFR, outperforming AstroCLIP fine-tuned (0.88 and 0.69) and general VLMs (R2 below 0.12 for GPT-4o, Claude-3.5, Llama-3.2 on property estimation).
- Morphology F1 improves by +0.17 on BAR (0.71) and SAC (0.79) versus the prior best, though SMH drops by 0.07.
- Expert analysis shows Euclidean FFN dominates property estimation (>40%) and hyperbolic FFN is highest on BAR, SPR, SAC (>35%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add geometry-aware prompts and geometry adapters | Qwen2-VL-2B-Instruct · Galaxy property estimation (sSFR) | R2: +0.15 | Helped |
| Add geometry-aware prompts and geometry adapters | Qwen2-VL-2B-Instruct · Galaxy morphology classification (BAR) | F1: +0.17 | Helped |
| Add geometry-aware prompts and geometry adapters | Qwen2-VL-2B-Instruct · Galaxy morphology classification (SAC) | F1: +0.17 | Helped |
| Add geometry-aware prompts and geometry adapters | Qwen2-VL-2B-Instruct · Galaxy morphology classification (SMH) | F1: −0.07 | Hurt |
| Add image modality (ablation) | Galaxy Walker · Galaxy property estimation | R2: −0.08 | Helped |
| Add spectrum modality (ablation) | Galaxy Walker · Galaxy property estimation (sSFR) | R2: −0.15 | Helped |
| Add spectrum modality (ablation) | Galaxy Walker · Galaxy property estimation (ZMW) | R2: −0.13 | Helped |
| Add Euclidean graph modality (ablation) | Galaxy Walker · Galaxy property estimation (M∗) | R2: −0.11 | Helped |
| Add hyperbolic graph modality (ablation) | Galaxy Walker · Galaxy property estimation | R2: −0.06 to −0.09 | Helped |
| Add spherical graph modality (ablation) | Galaxy Walker · Galaxy property estimation | R2: −0.06 to −0.09 | Helped |
| Use Geometry Adapter every 2 layers instead of every 4 layers (ablation) | Galaxy Walker · Galaxy morphology classification (BAR) | F1: BAR: 0.7 | Helped |
| Use Geometry Adapter every 2 layers instead of every 4 layers (ablation) | Galaxy Walker · Galaxy morphology classification (BLG) | F1: BLG: 0.8 | Helped |
Surprising findings
- Llama-3.2-90B-Vision-Instruct gets R2 of -2.99 on M* and -5.46 on ZMW, far worse than predicting the mean.
- Galaxy Walker's SMH F1 drops by 0.07 relative to the best prior result.
Limitations
- Only a small 2B VLM is tested; the authors note scaling to larger VLMs is unexplored.
- Only a limited number of geometric experts (three) is used.
- The paper's geometric motivations (e.g., hyperbolic space for accelerated expansion, spectra aligning with hyperbolic space) are interpretive and not directly tested.
Experimental setup
Datasets: DESI Legacy Imaging Surveys DR9 DESI EDR spectra Galaxy Zoo DECaLS
Benchmarks: galaxy property estimation (M*, ZMW, tage, sSFR) Galaxy Zoo DECaLS morphology classification (10 features)
Metrics: R2 F1
Baselines: Photometry Stein AstroCLIP (zero-shot) AstroCLIP (fine-tuned) GPT-4o Claude-3.5 Llama-3.2-90B-Vision-Instruct
Builds on: Qwen2-VL AstroCLIP GraphSAGE Riemannian GraphSAGE
Closest papers by meaning
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.84)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.84)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.84)
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.84)
- OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs (NeurIPS 2026 · similarity 0.84)
- HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.