Research map / Label-efficient and robust learning / Recognition and long-tail
Exploring Weight Balancing on Long-Tailed Recognition Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why weight balancing works for long-tailed recognition by decomposing it into weight decay/cross-entropy effects on Fisher's discriminant ratio (first stage) and implicit logit adjustment (second stage), and proposes a simpler one-stage method using WD, feature regularization, an ETF classifier and post-hoc logit adjustment.
Problem
Weight balancing (WB) performs strongly on long-tailed recognition, but it was unclear why it is effective; the paper seeks to explain its mechanism and simplify it.
Approach
Decomposes WB into its components and shows that the first stage (WD + CE) raises Fisher's discriminant ratio and suppresses the cone effect, while the second stage (WD + CB) acts as multiplicative logit adjustment; these insights allow a single-stage method (WD, feature regularization, ETF classifier, post-hoc multiplicative LA) that outperforms WB.
Key results
- On CIFAR100-LT (rho=100), the proposed one-stage WD&FR&ETF with multiplicative LA reaches 53.0% average accuracy, versus 50.6% for WB.
- Adding WD to CE raises training FDR on CIFAR100-LT from 1.28e2 (CE w/o WD) to 2.87e4.
- Multiplicative LA improves accuracy more than additive LA across the tested configurations on CIFAR100-LT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multiplicative logit adjustment after WD, FR, and ETF one-stage feature training (ablation) | WD&FR&ETF on CIFAR100-LT · CIFAR100-LT | Average accuracy (%): $47.6_{\pm0.5}$ -> $53.0_{\pm0.3}$ | Helped |
| Use multiplicative rather than additive logit adjustment (ablation) | WD&FR&ETF on CIFAR100-LT · CIFAR100-LT | Average accuracy (%): $50.9_{\pm0.3}$ -> $53.0_{\pm0.3}$ | Helped |
| Add WD to CE training (ablation) | CE feature training · CIFAR100-LT | Test FDR: $4.16 \times 10^1$ -> $1.07 \times 10^2$ | Helped |
| Add WD to CB training (ablation) | CB feature training · CIFAR100-LT | Test FDR: $2.42 \times 10^1$ -> $6.74 \times 10^1$ | Helped |
| Use CB instead of CE alone (ablation) | Feature training on CIFAR100-LT · CIFAR100-LT | Average accuracy (%): $38.5_{\pm0.6}$ -> $26.1_{\pm0.4}$ | Hurt |
| Add feature regularization to WD&ETF (ablation) | WD&ETF on CIFAR100-LT · CIFAR100-LT | Test FDR: $1.13 \!\times\!\! 10^2_{\pm0.1 \times 10^1}$ -> $1.22 \!\times\!\! 10^2_{\pm0.1 \times 10^1}$ | Helped |
| Omit logit adjustment (ablation) | WD&FR&ETF on CIFAR100-LT · CIFAR100-LT | Average accuracy (%): $53.0_{\pm0.3}$ vs $47.6_{\pm0.5}$ | Hurt |
| Replace CE with dot-regression loss (ablation) | WD&ETF on CIFAR100-LT · CIFAR100-LT | Test FDR: $1.13 \!\times\!\! 10^2_{\pm0.1 \times 10^1}$ -> $9.46 \!\times\!\! 10^1_{\pm0.8 \times 10^0}$ | Hurt |
| Replace CE with dot-regression loss (ablation) | WD&ETF on CIFAR100-LT · CIFAR100-LT | Average accuracy (%): $46.9_{\pm0.2}$ -> $46.0_{\pm0.4}$ | Hurt |
| Fix BN scaling parameters to a common small value and shifting parameters to 0 (ablation) | WD applied only to convolution layers · CIFAR100-LT | Test FDR: $7.95 \times 10^1$ -> $1.05 \times 10^2$ | Helped |
| Pass features through one randomly initialized linear layer and ReLU (ablation) | WD-trained ResNet34 features · CIFAR100-LT | FDR: $1.07 \times 10^2$ -> $1.13 \times 10^2$ | Helped |
Surprising findings
- WB does not make the classifier weight norm larger for Few classes on CIFAR10-LT, so the implicit logit adjustment equivalence fails when the class count is small.
- Passing features through a randomly initialized linear layer and ReLU often increases FDR for models trained with weight decay.
Limitations
- Experiments limited to MLP, ResNet and ResNeXt; generality to models such as ViT is unknown.
- Theoretical equivalence of second stage to logit adjustment requires assumptions (mu=0, NC occurred, large C or rho) that may not hold for small class counts.
Experimental setup
Datasets: CIFAR10 CIFAR100 mini-ImageNet ImageNet Helena MNIST
Benchmarks: CIFAR10-LT CIFAR100-LT mini-ImageNet-LT ImageNet-LT Helena
Metrics: top-1 accuracy (Many/Medium/Few/average) Fisher's discriminant ratio cosine similarity
Baselines: weight balancing class-balanced loss additive logit adjustment multiplicative logit adjustment dot-regression loss
Builds on: weight balancing class-balanced loss logit adjustment neural collapse
Closest papers by meaning
- Long-tailed Recognition with Model Rebalancing (NeurIPS 2025 · similarity 0.90)
- Difficulty-aware Balancing Margin Loss for Long-tailed Recognition (AAAI 2025 · similarity 0.89)
- Inverse Weight-Balancing for Deep Long-Tailed Learning (AAAI 2024 · similarity 0.89)
- Long-Tailed Learning as Multi-Objective Optimization (AAAI 2024 · similarity 0.89)
- Long-Tailed Recognition via Information-Preservable Two-Stage Learning (NeurIPS 2025 · similarity 0.87)
- Decoupled Optimisation for Long-Tailed Visual Recognition (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.