Research map / Label-efficient and robust learning / Recognition and long-tail

Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement

ICLR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Loss design and classifier theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Few-Class Arena (FCA), a benchmark and tool for evaluating efficient image classifiers in few-class settings (2-10 classes), plus a Silhouette-based similarity score (SimSS) that predicts few-class sub-model difficulty.

Problem

Models and rankings selected on many-class benchmarks (e.g., ImageNet-1K) do not reliably transfer to real-world applications with few classes (2-10), and there is no benchmark or difficulty measure for the few-class regime.

Approach

A unified few-class benchmark (FC-Full, FC-Sub, FC-Sim) that builds few-class subsets by configuration rather than copying data, and a Similarity-Based Silhouette Score (SimSS) computed from CLIP or DINOv2 features that correlates strongly (r≈0.88-0.90) with sub-model accuracy as a dataset difficulty proxy.

Key results

• SimSS computed with CLIP and DINOv2 correlates highly with few-class sub-model accuracy (r=0.90 CLIP, r=0.88 DINOv2) but poorly with full-model accuracy (r=0.18 and 0.26).
• On CIFAR100 with N_CL=2, sub-models reach 95.3-96.8% top-1 accuracy versus 70.6-78.0% for full ImageNet-trained-style full models on the same few-class evaluation setting (ResNet50/ResNet18/ViT-B table).
• No single model is best across the ten datasets; ResNet50 ranks 7th on ImageNet1K but 1st on Quickdraw345 among the ten models.

Measured effects reported in the paper

Surprising findings

• Full-class models do not reliably gain accuracy as the number of classes shrinks, and larger models do not consistently outperform smaller ones for sub-models in the few-class regime, contradicting the usual scaling intuition.
• Model rankings on ImageNet1K are poor predictors of rankings on other datasets, so many-class benchmark leaderboards may mislead model selection.

Limitations

• CLIP and DINOv2 were trained on natural internet images, so similarity may not transfer to sketches or texture-only datasets such as Quickdraw and Textures.
• Fine-tuning comparisons are limited to a few architectures and a single CIFAR100 setup.
• Full-model FC-Full results rely on pretrained weights only for ImageNet1K; other datasets were trained from scratch with minimal pipeline tuning.

Experimental setup

Datasets: ImageNet1K CIFAR100 CalTech101 CalTech256 CUB200 Food101 GTSRB43 Indoor67 Quickdraw345 Textures47

Benchmarks: ImageNet1K CIFAR100 CalTech101 CalTech256 CUB200 Food101 GTSRB43 Indoor67 Quickdraw345 Textures47

Metrics: Top-1 accuracy Pearson correlation coefficient standard deviation

Baselines: ResNet50 full-class model VGG16 ConvNeXt V2 Swin Transformer V2 ViT-Base

Builds on: MMPreTrain Silhouette Score CLIP DINOv2

Closest papers by meaning

• A Closer Look at Few-shot Classification (ICLR 2019 · similarity 0.88)
• Hard-Meta-Dataset++: Towards Understanding Few-Shot Performance on Difficult Tasks (ICLR 2023 · similarity 0.88)
• Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.86)
• Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.86)
• SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.86)
• Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.86)