Research map / Label-efficient and robust learning / Recognition and long-tail
Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HeroLT, a long-tailed learning benchmark and toolbox with 18 algorithms, 10 metrics, and 17 datasets across 6 tasks and 4 data modalities, plus a new Pareto-LT Ratio long-tailedness metric.
Problem
Long-tailed learning lacks systematic benchmarking across data long-tailedness, data complexity (tabular, sequential, grid, relational), and heterogeneous tasks, making algorithm selection and fair comparison difficult.
Approach
Organizes long-tailed benchmarking along three angles (long-tailedness metrics including a new Pareto-LT Ratio capturing both imbalance and number of categories, data complexity across modalities, and task heterogeneity) and provides a unified open-source toolbox evaluating 18 existing algorithms.
Key results
- PaCo reaches 58.3% overall accuracy on ImageNet-LT, and MiSLAS reaches 71.6% on iNatural 2018.
- Tail-GNN achieves 63.8% accuracy on Cora-Full, and it runs out of memory on Amazon-Electronics.
- No single algorithm consistently wins across tasks and datasets, so the authors argue algorithm selection depends on the scenario.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use upsampling with SMOTE instead of downsampling with NearMiss | Object recognition on Abalone · Abalone | balanced accuracy: 85.43% | Helped |
| Use XR-Transformer instead of the second-best method | Multi-label text classification · Eurlex-4K | Acc@1: 1.37% | Helped |
| Use X-Transformer instead of the suboptimal method | Multi-label text classification · AmazonCat-13K | Acc@5: 1.03% | Helped |
| Use recursive transformer learning instead of linear methods | Multi-label text classification · Eurlex-4K, AmazonCat-13K, and Wiki10-31K | accuracy | Helped |
| Use XR-Linear instead of XR-Transformer | Multi-label text classification · Eurlex-4K, AmazonCat-13K, and Wiki10-31K | inference time: more than 100x faster | Helped |
| Use PaCo instead of the suboptimal method | Image classification · ImageNet-LT | overall accuracy: 16.37% | Helped |
| Use PaCo instead of other methods | Image classification · Places-LT | few-shot accuracy: 27.15% | Helped |
| Use MiSLAS instead of other methods | Image classification · Places-LT | few-shot accuracy: 14.90% | Helped |
| Increase the imbalance factor of the semi-synthetic datasets | Long-tailed image classification methods · CIFAR-10-LT and CIFAR-100-LT | accuracy | Hurt |
| Use Tail-GNN instead of the second-best method | Node classification · Cora-Full | accuracy: 4.93% | Helped |
| Adjust the number of classes considered as tail in ImGAGN | Node classification · Relational datasets | accuracy | Helped |
| Use SmoteR instead of SMOGN | Long-tailed regression · Housing | MAE | Helped |
Surprising findings
- No single long-tailed algorithm statistically outperforms the others across all tasks and domains.
- ImGAGN performs poorly on datasets with many categories because it treats only one class as tail by default.
Limitations
- Tail-GNN runs out of memory on the largest dataset, Amazon-Electronics.
- Long-tailedness metrics do not comprehensively capture all dataset characteristics, per the authors' own analysis.
- Benchmark covers only 18 algorithms and is described as still being updated.
Experimental setup
Datasets: Glass Abalone Housing EURLEX-4K AMAZONCat-13K Wiki10-31K ImageNet-LT Places-LT iNatural 2018 CIFAR 10-LT CIFAR 100-LT LVIS v0.5
Benchmarks: Glass Abalone EURLEX-4K AMAZONCat-13K Wiki10-31K ImageNet-LT Places-LT iNatural 2018 CIFAR 10-LT CIFAR 100-LT LVIS v0.5 Cora-Full
Metrics: Acc bAcc precision recall MAP MAE MSE Pearson GM running time
Baselines: SMOTE NearMiss X-Transformer XR-Transformer XR-Linear OLTR BALMS TDE Decoupling BBN MiSLAS PaCo
Builds on: SMOTE decoupling BBN PaCo XR-Transformer
Closest papers by meaning
- Breaking Long-Tailed Learning Bottlenecks: A Controllable Paradigm with Hypernetwork-Generated Diverse Experts (NeurIPS 2024 · similarity 0.88)
- Long-Tailed Learning as Multi-Objective Optimization (AAAI 2024 · similarity 0.87)
- Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers (NeurIPS 2025 · similarity 0.86)
- Learning Only When It Matters: Cost-Aware Long-Tailed Classification (AAAI 2024 · similarity 0.85)
- SSE-SAM: Balancing Head and Tail Classes Gradually Through Stage-Wise SAM (AAAI 2025 · similarity 0.85)
- LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.