Research map / Label-efficient and robust learning / Recognition and long-tail
What Would Gauss Say About Representations? Probing Pretrained Image Models using Synthetic Gaussian Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynBench, a task-agnostic, data-free evaluation of pretrained image model representations that compares robustness-accuracy trade-offs on synthetic Gaussian mixture data against a Bayes-optimal theoretical reference, yielding a SynBench-Score.
Problem
Pretrained image model representations are usually evaluated on many real downstream datasets, which is costly, privacy-sensitive, and dependent on biased evaluation data; a task-agnostic quality measure is needed.
Approach
Use binary class-conditional Gaussian mixtures whose optimal robust linear classifier is known analytically to derive a robustness-accuracy reference; compare pretrained representations of the synthetic samples to this reference via the ratio of area-under-curves (SynBench-Score).
Key results
- SynBench-Score correlates with average downstream linear-probing accuracy at Pearson >0.9 when using over 30k synthetic samples, versus 0.79 at 2k samples (baselines 0.46-0.55 at 2k).
- SynBench-guided epsilon-robust linear probing on ViT-B/16 with epsilon=0.2 improves TinyImageNet standard and robust accuracy by +0.7% and +2.5%.
- Using a heptadiagonal covariance (Gaussian-H) closer to real data flips the ViT-B/16 vs ViT-L/16 ranking to match real downstream results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune ViT-B/16-in21k on ImageNet-1k | ViT-B/16-in21k · 27 downstream tasks | average accuracy | Helped |
| Use ViT-B/16 instead of ViT-B/16-in21k | Pretrained representations · Class-conditional Gaussian | SynBench-Score: 0.20 -> 0.33 | Helped |
| Increase model size from ViT-Ti/16 to ViT-B/16 or ViT-L/16 | ViT representations · Class-conditional Gaussian | SynBench-Score: 0.01 -> 0.33; 0.01 -> 0.26 | Helped |
| Use self-supervised DINO training instead of supervised training | ViT-B/16 · Class-conditional Gaussian | SynBench-Score: 0.33 -> 0.55 | Helped |
| Use ϵ = 0.2 instead of standard linear probing (ϵ = 0) (ablation) | ViT-B/16 · TinyImagenet | standard accuracy: +0.7% | Helped |
| Use ϵ = 0.2 instead of standard linear probing (ϵ = 0) (ablation) | ViT-B/16 · TinyImagenet | robust accuracy: +2.5% | Helped |
| Use heptadiagonal covariance (Gaussian-H) instead of identity covariance (Gaussian-I) (ablation) | SynBench · Class-conditional Gaussian | SynBench-Score comparison between ViT-L/16 and ViT-B/16: ViT-B/16: 0.18; ViT-L/16: 0.18; at ϵ = 0.2: 0.24; 0.28 | Mixed |
| Use a more downstream-like covariance prior (ablation) | SynBench · CIFAR10, SVHN, and TinyImageNet | linear-probing standard and robust accuracy | Helped |
| Evaluate with more synthetic samples (ablation) | SynBench · 27 downstream tasks | Pearson correlation with average accuracy: 0.79; 0.89; > 0.9 | Helped |
| Use SynBench instead of the reported task-agnostic baselines | Pretrained model selection · 27 downstream tasks | Pearson correlation with average accuracy: 0.79 | Helped |
| Evaluate with SynBench instead of baselines | Pretrained model selection · CIFAR10 adversarial robustness | Pearson correlation with CW attack results: 0.74 to 0.84 | Helped |
Surprising findings
- SynBench-Score with identity covariance ranks ViT-B/16 above ViT-L/16, contrary to actual downstream accuracy, showing sensitivity to the synthetic data prior.
- SynBench-Score is not monotonic in epsilon, so standard linear probing (epsilon=0) may not be the best probing choice.
Limitations
- Relies on Gaussian data assumption; mismatch with real data can change rankings (e.g., identity covariance).
- Robustness is measured only in l2 adversarial perturbation with linear classifier reference.
- Evaluation limited to image classification pretrained models (ViT, ResNet variants).
Experimental setup
Datasets: synthetic class-conditional Gaussian data CIFAR10 SVHN TinyImageNet 27 downstream tasks (CLIP benchmark)
Benchmarks: 27 downstream image classification tasks (average accuracy) CIFAR10 TinyImageNet SVHN ImageNet-C ImageNet-A
Metrics: SynBench-Score Pearson correlation standard accuracy robust accuracy model selection consistency
Baselines: Val loss MDL SDL LogME SFDA ImageNet-C accuracy ImageNet-A accuracy
Builds on: Fisher's linear discriminant Bayes optimal robust classifier (Dan et al. 2020) epsilon-robust linear probing (Fan et al. 2021) LogME SFDA
Closest papers by meaning
- Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025 · similarity 0.85)
- Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing (NeurIPS 2026 · similarity 0.85)
- Enhancing Pre-trained Representation Classifiability can Boost its Interpretability (ICLR 2025 · similarity 0.85)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.85)
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.84)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.