Research map / Vision-language and multimodal / CLIP and zero-shot

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models

NeurIPS 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STS, a lightweight episodic test-time adaptation method for CLIP that learns a shared low-dimensional steering vector in the SVD subspace of text prototypes, minimizing marginal entropy over augmented views without backpropagating through encoders.

Problem

Zero-shot VLMs such as CLIP degrade under test-time distribution shifts, and existing episodic TTA methods (e.g., test-time prompt tuning) are costly because they backpropagate through large encoders or modify model components.

Approach

Compute an SVD of the initial class text prototypes, keep the top-k right singular vectors selected by Gavish-Donoho hard thresholding, and learn one shared per-sample coefficient vector that shifts all text prototypes within that subspace, optimized with entropy minimization over confidence-filtered augmented views.

Key results

• On ImageNet-A with CLIP ViT-B/16, STS reaches 61.23% vs 54.39% for TPT and 47.87% zero-shot; the ensemble variant reaches 64.29%.
• STS$_{\text{Ensemble}}$ averages 65.06% across ten fine-grained datasets vs 64.78% for TPT, with STS alone at 63.86%.
• STS is reported at 0.09 s per sample and 1.4 GB memory vs 0.75 s and 17.6 GB for TPT on ImageNet (ViT-B/16).

Measured effects reported in the paper

Surprising findings

• Per-class shift vectors give only marginal gains (0.03% average) over a single shared vector, so the simpler shared design is kept.
• On fine-grained tasks, STS underperforms TPT on EuroSAT by a large margin, which the authors attribute to EuroSAT being a known failure mode for TTA methods.

Limitations

• Steering is linear within the SVD subspace, so highly non-linear domain shifts may not be addressed.
• Compute scales linearly with the number of augmented views because each view needs a separate encoder forward pass.
• Performance is weaker on some datasets such as EuroSAT, where standard TPT is better.

Experimental setup

Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Flowers102 DTD OxfordPets UCF101 Caltech101 FGVCAircraft EuroSAT

Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Flowers102 DTD OxfordPets UCF101 Caltech101 FGVCAircraft EuroSAT

Metrics: top-1 accuracy testing time per sample memory usage (GB)

Baselines: Zero-Shot CLIP Ensemble CoOp TPT DiffTPT C-TPT TPS TTL MaPLe

Builds on: test-time prompt tuning CLIP Test-Time Prototype Shifting Gavish-Donoho optimal hard thresholding MaPLe

Closest papers by meaning

• Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.93)
• On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.91)
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
• Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.90)
• Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.90)
• The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.89)