Research map / Label-efficient and robust learning / Recognition and long-tail
Learning from Neighbors: Category Extrapolation for Long-Tail Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes category extrapolation for long-tail learning: LLM-queried fine-grained neighbor categories are web-crawled as auxiliary data, trained with a neighbor-silencing loss, and auxiliary classifier weights are masked at inference.
Problem
Long-tail visual recognition suffers from limited sample diversity and biased features for tail classes, with re-weighting and re-sampling bounded by existing known classes.
Approach
Increase dataset granularity by adding open-set fine-grained neighbor categories (proposed by an LLM and crawled from the web), train with a neighbor-silencing balanced softmax loss that down-weights neighbor-pair logits, and discard auxiliary classifier weights at test time.
Key results
- On ImageNet-LT with DINOv2 pre-training, the method reaches 82.0% overall accuracy versus 79.6% for the Bal-CE baseline, with tail accuracy rising from 71.1% to 76.2%.
- Training from scratch on ImageNet-LT, overall accuracy improves from 60.9% to 68.2%, with tail accuracy rising 16.0 points (41.4% to 57.4%).
- On iNaturalist 2018 with DINOv2, overall accuracy improves from 85.0% to 87.0%, and on Places-LT with DINOv2 from 49.5% to 50.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use category extrapolation with neighbor-silencing loss instead of Bal-CE | DINOv2 long-tail classification · ImageNet-LT | Overall accuracy: 79.6 -> 82.0 | Helped |
| Use category extrapolation with neighbor-silencing loss instead of Bal-CE | DINOv2 long-tail classification · iNaturalist 2018 | Overall accuracy: 85.0 -> 87.0 | Helped |
| Use category extrapolation with neighbor-silencing loss instead of Bal-CE | DINOv2 long-tail classification · Places-LT | Overall accuracy: 49.5 -> 50.8 | Helped |
| Use random auxiliary categories instead of selected neighbor categories (ablation) | Bal-CE with DINOv2 on ImageNet-LT · ImageNet-LT | Overall accuracy: 79.6 -> 77.2 | Hurt |
| Add selected neighbor categories (ablation) | Bal-CE with DINOv2 on ImageNet-LT · ImageNet-LT | Overall accuracy: 79.6 -> 80.5 | Helped |
| Add neighbor-silencing loss to neighbor-category training (ablation) | Bal-CE with DINOv2 on ImageNet-LT · ImageNet-LT | Overall accuracy: 80.5 -> 81.4 | Helped |
| Use trained target-class classifier weights directly instead of retraining by linear probing (ablation) | Neighbor-silencing model with DINOv2 · ImageNet-LT | Overall accuracy: 81.4 -> 82.0 | Helped |
Surprising findings
- Randomly sampled auxiliary data significantly degrades performance during fine-tuning (e.g., DINOv2 ImageNet-LT overall 79.6% to 77.2%), while neighbor-selected categories help.
- Simply masking out auxiliary classifier weights at inference works better than re-training the classifier with linear probing.
Limitations
- Relies on LLM-proposed neighbor categories and web image search, which can be noisy and requires filtering with thresholds.
- Gains on head classes are small or absent; improvements concentrate on medium and few-shot classes.
- Auxiliary data volume and sampling ratio are sensitive hyperparameters; too many auxiliary samples overwhelm training.
Experimental setup
Datasets: ImageNet-LT iNaturalist 2018 Places-LT ImageNet-21k OpenImages
Benchmarks: ImageNet-LT iNaturalist 2018 Places-LT
Metrics: top-1 accuracy overall many-shot accuracy medium-shot accuracy few-shot accuracy
Baselines: Balanced Softmax (Bal-CE) Remix LiVT LIFT classifier re-training (cRT/LA) Random auxiliary data (RD)
Builds on: Balanced Softmax LiVT LIFT DINOv2 CutMix
Closest papers by meaning
- DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge (NeurIPS 2024 · similarity 0.87)
- Feature Fusion from Head to Tail for Long-Tailed Visual Recognition (AAAI 2024 · similarity 0.86)
- Long-Tailed Learning as Multi-Objective Optimization (AAAI 2024 · similarity 0.86)
- LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups (NeurIPS 2025 · similarity 0.86)
- Pursuing Better Decision Boundaries for Long-Tailed Object Detection via Category Information Amount (ICLR 2025 · similarity 0.86)
- LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.