Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Improving Generalization via Meta-Learning on Hard Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOLERE, which jointly optimizes the train/validation split and a learned-reweighting classifier so that hard-to-classify instances serve as validation data, improving generalization over ERM and reweighting baselines.
Problem
Learned reweighting (LRW) methods depend on the choice of validation set, yet this choice is usually fixed or clean; the paper asks how to optimize the validation set to maximize classifier generalization.
Approach
Formalize validation-set selection as a tri-level problem (MOLERE), reduce it to a bi-level min-max game between a "splitter" network that finds hard samples and a meta-network that learns instance weights, and use a train-twice margin heuristic (LRW-Hard/Easy/Random) to show that hard validation sets help.
Key results
- On ImageNet-1K with ViT-B/16, LRW-Hard reaches 85.09% vs 84.11% ERM and 83.17% LRW-Easy (about 1% over ERM; 1.92-point gap to Easy).
- On Inst. C-10 noisy labels, LRWOpt reaches 71.87% vs 70.34% for the best baseline (MAPLE), a 1.53-point gain.
- Using ImageNet-R or ImageNet-A as LRW validation improves in-domain ImageNet test accuracy by about 0.5-0.7 points and OOD accuracy by 1.7-3 points over ERM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hard rather than easy validation samples in LRW (ablation) | LRW classifiers · ImageNet-1K | accuracy: 83.17 -> 85.09 | Helped |
| use hard rather than random validation samples in LRW (ablation) | LRW classifiers · ImageNet-1K | accuracy: 83.87 -> 85.09 | Helped |
| use hard rather than easy validation samples (ablation) | LRW classifiers · ImageNet-1K | accuracy: 1.92% | Helped |
| optimize the validation split end-to-end with LRWOpt instead of using ERM | ImageNet-1K with ViT-B/16 pretrained · ImageNet-1K | accuracy: 84.11 -> 84.94 | Helped |
| use LRWOpt rather than the strongest reported reweighting baseline | CIFAR-100 class-skew training · CIFAR-100, class skew 200 | accuracy: 40.36 ± 1.0 -> 42.33 ± 0.8 | Helped |
| use LRWOpt rather than GDW | CIFAR-100 class-skew training · CIFAR-100, class skew 50 | accuracy: 48.89 ± 0.8 -> 50.77 ± 0.7 | Helped |
| use LRWOpt rather than GDW | CIFAR-100 class-skew training · CIFAR-100, class skew 10 | accuracy: 61.67 ± 0.5 -> 63.28 ± 0.8 | Helped |
| use LRWOpt rather than FSR | CIFAR-100 class-skew training · CIFAR-100, class skew 1 | accuracy: 74.68 ± 0.2 -> 75.12 ± 0.3 | Helped |
| use LRWOpt rather than GDW | instance-dependent noisy-label learning · Inst. C-10 | accuracy: 69.12 -> 71.87 | Helped |
| use LRWOpt rather than GDW | instance-dependent noisy-label learning · Clothing-1M | accuracy: 73.12 -> 73.97 | Helped |
| use IN-R as LRW validation data rather than train with ERM on ImageNet-1K | ImageNet classifier · IN-R test | accuracy: 46.1 -> 49.1 | Helped |
| add IN-R data to ERM training rather than use ERM on ImageNet-1K alone (ablation) | ImageNet classifier · ImageNet-1K test | accuracy: 75.65 -> 74.89 | Hurt |
Surprising findings
- Simply adding OOD data (IN-R) to the ERM training set degrades in-domain ImageNet accuracy while LRW with the same data as validation improves it.
- Easy validation sets make LRW perform worse than plain random validation, and the paper's simple margin heuristic is not competitive under very high label noise.
Limitations
- The margin-based train-twice heuristic is an inexact hardness proxy and is not competitive under very high label noise.
- Gains over ERM are modest (about 1 point on ImageNet), and LRW-Hard is nominally below LRWOpt or noisy on some datasets.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 ImageNet-100 Aircraft Stanford Cars Oxford-IIIT Pet Diabetic Retinopathy Camelyon iWildCam Inst. C-10 Clothing-1M ImageNet-A
Benchmarks: ImageNet-1K CIFAR-100 ImageNet-100 Clothing-1M Inst. C-10 Camelyon iWildCam Diabetic Retinopathy (APTOS) ImageNet-A ImageNet-R
Metrics: top-1 accuracy accuracy gain over ERM margin
Baselines: ERM MWN FSR L2R MAPLE GDW BiLAW StableNet MBR Rho-Loss
Builds on: learned reweighting (L2R) Meta-Weight-Net Fast Sample Reweighting Just Train Twice Margin-based reweighting
Closest papers by meaning
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (ICLR 2020 · similarity 0.89)
- On the Stability and Generalization of Meta-Learning: the Impact of Inner-Levels (NeurIPS 2025 · similarity 0.88)
- Meta–Gradient Boosted Decision Tree Model for Weight and Target Learning (ICML 2016 · similarity 0.88)
- MetaReg: Towards Domain Generalization using Meta-Regularization (NeurIPS 2018 · similarity 0.87)
- Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling (AAAI 2025 · similarity 0.87)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.