Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models

ICLR 2025 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Transductive and semantic-dictionary zero-shot · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks 10 vision-language foundation models (66 backbones, 15 datasets) under low-resolution zero-shot classification, proposes WAR robustness metric, and LR-TK0, which adds trainable low-resolution tokens to frozen ViTs distilled on synthetic diffusion images.

Problem

Vision-language foundation models are evaluated mostly at high resolution, and their zero-shot robustness to low-resolution or pixelated images is underexplored; existing robustness metrics (relative robustness and simple averaging) are misleading near random accuracy and overlook some datasets.

Approach

Benchmark zero-shot robustness across many FM backbones at 16-128 px, introduce a random-chance-aware relative robustness and dataset-weighted aggregation (WAR), and improve robustness by inserting trainable LR tokens before each frozen transformer block, trained via self-supervised distillation from the frozen HR teacher on PixArt-alpha synthetic images.

Key results

• LR-TK0 improves EVA-B/16 at 16x16 from 30.7 to 35.4 WAR and 38.0 to 42.4 SAR with about 3% more parameters (155.2M vs 149.7M).
• On MetaCLIP-B/16 at 16x16, LR-TK0 raises WAR from 27.2 to 38.9 and Acc from 23.4 to 30.2.
• The paper reports larger models and higher-quality pre-training data (DataComp-1B over LAION-2B) improve LR robustness, while fine-tuned and 336px models are less robust.

Measured effects reported in the paper

Surprising findings

• Random-guess-level models can obtain high relative robustness (around 20-40%) when the top-resolution accuracy is also near chance, motivating the WAR correction.
• CLIP ResNets are markedly less robust to low resolution than ViTs.

Limitations

• Zero-shot super-resolution and diffusion-based SR baselines are too expensive to evaluate on large datasets such as ImageNet.
• WAR dataset weights are optimized on ranking correlations and may not represent true robustness; some weights hit bounds (0.01 or 1.0).
• Dataset-specific robustness differences are left as future work.

Experimental setup

Datasets: Conceptual Captions ImageNet ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch Caltech101 DTD Food101 SUN397 Stanford Cars FGVC Aircraft

Benchmarks: ImageNet ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch Caltech101 DTD Food101 SUN397 Stanford Cars FGVC Aircraft Oxford Pets

Metrics: top-1 accuracy SAR WAR relative robustness

Baselines: SAR-based robustness BSRGAN ESRGAN SwinIR AddSR Inf-DiT IDM VPT RobustSAM

Builds on: CLIP EVA-CLIP visual prompt tuning CLIP evaluation protocol

Closest papers by meaning

• Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.89)
• Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.88)
• Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models (ICLR 2024 · similarity 0.87)
• Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input (AAAI 2026 · similarity 0.87)
• LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.87)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.87)