Research map / Vision-language and multimodal / Cross-modal learning

Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ECHO-k, a label-free, task-agnostic RL policy that sequentially acquires views so the partial representation of a frozen foundation model approaches its full-view representation, with a linear-setting regret analysis.

Problem

Test-time sequential feature/modality acquisition under a budget when the downstream task is unknown and no labels are available for training the acquisition policy.

Approach

Use the frozen pretrained model's full-view latent representation as a self-supervised target; reward an RL policy (PPO with invalid-action masking) for marginal reduction in representation error, and give theory bounding latent regret vs. an oracle greedy policy under a Gumbel/Plackett-Luce latent model and near-isometry assumption.

Key results

• On MiniBooNE, ECHO-k reaches 81.70% accuracy at 20% budget vs. 79.80% for the best adaptive baseline (CIG).
• On hPancreas with 20% reference masking, ECHO-k reaches accuracy 0.808 at 20% budget vs. 0.679 for Variance (reported in robustness table).
• On MMEarth soil pH, ECHO-k outperforms non-oracle baselines reaching strong R^2 after acquiring 20-30% of views.

Surprising findings

• Masked reference representations outperform the full-view reference on MNIST-14 at 20% and 40% budgets and on hPancreas at 20%.
• ECHO-k slightly underperforms in the first ~2-3% of acquisitions on hPancreas before recovering.

Limitations

• Theory holds only in a stylized linear model with Gumbel latent correlation and near-isometry assumptions; the bound is on proxy and linear-model regret.
• Acquisition costs and constraints are domain-specific; cost-aware extension requires a cost model.
• Does not test explicit distribution shift between reference-construction data and deployment domain.

Experimental setup

Datasets: hPancreas NSCLC imaging mass cytometry cohort MMEarth-Bench MiniBooNE MNIST Fashion-MNIST

Benchmarks: hPancreas NSCLC cancer subtyping and grading MMEarth-Bench soil pH MiniBooNE MNIST Fashion-MNIST

Metrics: accuracy R^2 MSE normalized representation recovery

Baselines: Oracle Random Variance GIR Laplacian Score SPEC MCFS UDFS SVD-Ent ESM2-based orderings GSMRL-AIR CIG

Builds on: PPO Plackett-Luce model DINOv2 invalid action masking

Closest papers by meaning

• Attention Bootstrapping for Multi-Modal Test-Time Adaptation (AAAI 2025 · similarity 0.85)
• Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data (ICML 2025 · similarity 0.85)
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.85)
• MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.84)
• Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.84)
• Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.84)