Research map / Large language models / Instruction and fine-tuning
LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LTGC, which uses LMMs (GPT-4V), an LLM (GPT-4), and DALL-E to generate diverse tail-class images with self-reflection and CLIP-based iterative evaluation, then fine-tunes CLIP ViT-B/32 with LoRA using a BalanceMix mixup-plus-balanced-sampling scheme for long-tail recognition.
Problem
Long-tail recognition suffers from scarce and low-diversity tail-class data, and existing CLIP-based transfer and augmentation methods struggle to supply correct, diverse knowledge for tail classes.
Approach
Use LMMs to describe existing tail images with templates, use an LLM to extend these descriptions with missing features and scenes (with number- and repetition-checking self-reflection), generate images via a text-to-image model, filter and refine low-quality images using CLIP similarity feedback, and fine-tune CLIP with a BalanceMix that combines balanced sampling and Mixup across original and ge
Key results
- LTGC reaches 80.6% overall top-1 on ImageNet-LT, 3.4 points above VL-LTR (77.2%).
- LTGC reaches 82.5% overall on iNaturalist 2018, 2.3 points above RAC (80.2%).
- On Places-LT, LTGC reaches 54.1% overall and 52.1% few-shot, versus LPT's 50.1% overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LTGC rather than VL-LTR | CLIP-based long-tail recognition · ImageNet-LT | Overall accuracy: 77.2 -> 80.6 | Helped |
| Use LTGC rather than LPT | CLIP-based long-tail recognition · Places-LT | Overall accuracy: 50.1 -> 54.1 | Helped |
| Use LTGC rather than RAC | CLIP-based long-tail recognition · iNaturalist 2018 | Overall accuracy: 80.2 -> 82.5 | Helped |
| Add BalanceMix instead of directly fine-tuning on generated and original data (ablation) | LTGC · ImageNet-LT | Overall accuracy: 58.3 -> 80.6 | Helped |
| Add BalanceMix instead of directly fine-tuning on generated and original data (ablation) | LTGC · iNaturalist 2018 | Overall accuracy: 69.5 -> 82.5 | Helped |
| Replace balanced sampling with BalanceMix (ablation) | LTGC · ImageNet-LT | Overall accuracy: 63.9 -> 80.6 | Helped |
| Replace balanced sampling with BalanceMix (ablation) | LTGC · iNaturalist 2018 | Overall accuracy: 73.8 -> 82.5 | Helped |
| Add balanced sampling to Mixup as in BalanceMix (ablation) | LTGC · ImageNet-LT | Overall accuracy: 73.4 -> 80.6 | Helped |
| Add balanced sampling to Mixup as in BalanceMix (ablation) | LTGC · iNaturalist 2018 | Overall accuracy: 75.2 -> 82.5 | Helped |
Surprising findings
- Directly querying LMMs such as GPT-4 for the class label performs far worse than LTGC (72.1 vs 80.6 on ImageNet-LT), despite LMMs describing images well.
- Simple balanced sampling or Mixup alone on generated data lowers accuracy relative to the full BalanceMix (63.9 and 73.4 vs 80.6 on ImageNet-LT).
Limitations
- Relies on closed commercial models (GPT-4V, GPT-4, DALL-E) whose cost and reproducibility are not addressed.
- Reported baseline numbers are taken from original papers rather than re-run, and the generated-data size is not specified.
- Domain gap between generated and original images remains, which the paper only partly addresses.
Experimental setup
Datasets: ImageNet-LT Places-LT iNaturalist 2018
Benchmarks: ImageNet-LT Places-LT iNaturalist 2018
Metrics: top-1 accuracy many-shot accuracy medium-shot accuracy few-shot accuracy
Baselines: CLIP Zero-Shot CLIP Finetune VL-LTR LPT RAC RIDE PaCo MDCS LENS MiniGPT4
Builds on: CLIP Mixup LPT VL-LTR
Closest papers by meaning
- DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge (NeurIPS 2024 · similarity 0.88)
- Learning from Neighbors: Category Extrapolation for Long-Tail Learning (CVPR 2025 · similarity 0.86)
- Long-tailed Recognition with Model Rebalancing (NeurIPS 2025 · similarity 0.86)
- Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition (ICML 2024 · similarity 0.86)
- Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition (NeurIPS 2024 · similarity 0.86)
- Feature Fusion from Head to Tail for Long-Tailed Visual Recognition (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.