Research map / Label-efficient and robust learning / Recognition and long-tail
Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Focal-SAM, which weights class-wise sharpness with focal weights (1-π_i)^γ inside a single SAM-style update, giving fine-grained head/tail flatness control at about SAM-level cost, plus a PAC-Bayes bound of order 1/n.
Problem
Long-tailed classification where SAM variants either sharpen head-class landscapes (ImbSAM) or cost roughly C extra backpropagations per step (CC-SAM), leaving a trade-off between control and efficiency.
Approach
Replace per-class perturbations of CC-SAM and the head/tail split of ImbSAM with a single focal-weighted class-wise sharpness penalty (1-π_i)^γ, optimized with three backpropagations per step, and derive a PAC-Bayesian bound with a 1/n rate.
Key results
- On CIFAR-100 LT IR100, LA+Focal-SAM reaches 50.7% all-class accuracy versus 50.0% for LA+SAM and 47.3% for LA+ImbSAM.
- On ImageNet-LT with LA loss, LA+Focal-SAM reaches 54.3% versus 53.6% for LA+SAM.
- Focal-SAM costs about 1.3-1.7x SAM per epoch versus 2-15x for CC-SAM (Table of training times).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Focal-SAM to LA loss during ResNet-50 training | LA baseline · ImageNet-LT | balanced accuracy (All): 53.6 -> 54.3 | Helped |
| Add Focal-SAM to LA loss during ResNet-50 training | LA baseline · iNaturalist | balanced accuracy (All): 71.0 -> 71.8 | Helped |
| Add Focal-SAM to FFT fine-tuning | FFT baseline · ImageNet-LT | balanced accuracy (All): 73.6 -> 73.9 | Helped |
| Add Focal-SAM to LIFT fine-tuning | LIFT baseline · iNaturalist | balanced accuracy (All): 79.8 -> 80.0 | Helped |
| Add Focal-SAM to FFT fine-tuning | FFT baseline · ImageNet-Sketch | balanced accuracy (All): 40.6 -> 41.0 | Helped |
| Add Focal-SAM to LIFT fine-tuning | LIFT baseline · ImageNetV2 | balanced accuracy (All): 67.7 -> 68.0 | Helped |
| Add Focal-SAM to FFT fine-tuning | FFT baseline · ImageNet-C | balanced accuracy (All): 45.6 -> 46.1 | Helped |
| Replace CC-SAM with Focal-SAM | ResNet SAM training · CIFAR-100 LT epoch training | training time: 19.70s (4.10$\times$) -> 6.71s (1.40$\times$) | Helped |
| Increase gamma from a moderate value to a larger value (ablation) | Focal-SAM · CIFAR-LT | accuracy | Mixed |
| Increase lambda from a moderate value to a larger value (ablation) | Focal-SAM · CIFAR-LT | accuracy | Mixed |
Surprising findings
- ImbSAM over-sharpens head classes, and excluding head classes from SAM hurts head performance even though it helps tail classes.
- Focal-SAM's gains on OOD ImageNet variants (about 0.5-4.3%) are larger than its in-distribution gains for SAM-based methods on fine-tuned foundation models.
Limitations
- Focal-SAM still costs about 50% more time per epoch than SAM.
- Theory assumes the loss is bounded and that the expected focal loss is nonnegative, which the authors state is a simplifying assumption.
- Performance depends on tuning gamma and lambda, and excessive gamma harms tail accuracy.
Experimental setup
Datasets: CIFAR-10 LT CIFAR-100 LT ImageNet-LT iNaturalist
Benchmarks: CIFAR-10 LT CIFAR-100 LT ImageNet-LT iNaturalist ImageNet-Sketch ImageNetV2 ImageNet-C
Metrics: balanced accuracy head/medium/tail accuracy
Baselines: SAM ImbSAM CC-SAM LDAM+DRW CB cRT DiVE DRO-LT DisAlign WB LIFT
Builds on: SAM ImbSAM CC-SAM LA loss VS loss LIFT
Closest papers by meaning
- SSE-SAM: Balancing Head and Tail Classes Gradually Through Stage-Wise SAM (AAAI 2025 · similarity 0.91)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.90)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.89)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.89)
- Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems (NeurIPS 2024 · similarity 0.88)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.