Research map / Vision-language and multimodal / Multimodal LLMs
The Perception–Physics Paradox: Probing Scientific Alignment with TC-Bench
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Perception–Physics Paradox and a structural-isomorphism probing framework, releasing TC-Bench (global tropical cyclone IR imagery benchmark) to show frozen vision foundation models collapse physically in intense cyclone regimes.
Problem
Vision foundation models can score well on perceptual and OOD benchmarks for satellite imagery while failing to encode physical state variables (e.g., tropical cyclone minimum central pressure) in extreme regimes, so average or perception-based OOD accuracy is a poor proxy for scientific utility.
Approach
Defines scientific alignment via structural isomorphism (latent space is a linear reparameterization of the physical state up to bounded residuals), derives necessary probe conditions (static fidelity, dynamic coherence, manifold consistency) with linear probes, and applies them to frozen VFMs on a regime-stratified cyclone dataset.
Key results
- Across frozen VFMs, static fidelity error and catastrophic outliers rise sharply once minimum central pressure falls below 980 hPa.
- Manifold inconsistency rises from about 20% in the moderate regime to about 55% in the intense regime.
- For DINOv3-base, effective dimensionality of within-bin features drops by roughly 60% in intense regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Intense regime (P_c<980 hPa) vs moderate regime for frozen VFM linear probe | frozen VFMs (DINOv2, DINOv3, CLIP, SigLIP, SigLIP2, MAE) · TC-Bench static fidelity | normalized static fidelity error / manifold inconsistency: manifold inconsistency rises from ~20% to ~55% | Hurt |
| Within-bin feature covariance effective dimensionality in intense regime | DINOv3-base · TC-Bench failure mode analysis | participation-ratio effective dimensionality: reduced by approximately 60% | Hurt |
| Nonlinear MLP and Transformer probes instead of linear probe (ablation) | frozen VFMs · TC-Bench static fidelity | static fidelity error | No clear effect |
Surprising findings
- A supervised pixel-space baseline trained from scratch achieves lower error in the intense regime than frozen VFMs, indicating the signal is present in imagery.
- Nonlinear MLP and transformer probes show the same moderate-to-intense gap, so the failure is not a linear-probe artifact.
Limitations
- Framework provides necessary but not sufficient conditions for scientific alignment.
- Primary analyses focus on minimum central pressure because wind reporting is inconsistent across agencies.
- Probes are restricted to linear readouts for the main results, with nonlinear checks only in appendix.
Experimental setup
Datasets: TC-Bench GridSat-B1 IBTrACS v4r01
Benchmarks: TC-Bench
Metrics: normalized static fidelity error dynamic coherence gap manifold inconsistency effective dimensionality
Baselines: Digital Typhoon pixel-space supervised baseline (ResNet)
Builds on: Digital Typhoon causal representation learning
Closest papers by meaning
- IDOL: Meeting Diverse Distribution Shifts with Prior Physics for Tropical Cyclone Multi-Task Estimation (NeurIPS 2025 · similarity 0.84)
- A systematic evaluation of vision-language models for observational astronomical reasoning tasks (NeurIPS 2026 · similarity 0.84)
- SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Models (NeurIPS 2026 · similarity 0.84)
- VESTA: Visual Exploration with Statistical Tool Agents (NeurIPS 2026 · similarity 0.83)
- Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.83)
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.