Research map / Architectures and efficiency / RNNs and normalization
Learning with Adaptive Resource Allocation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LARA, an online framework for scheduling multiple time-constrained model training tasks on a shared budget of compute, combining weighted least-squares loss-curve extrapolation, an optimal adaptive binary-tree allocation search, and explore-then-exploit balancing.
Problem
Training multiple deadline-constrained learning tasks concurrently under a budgeted compute throughput, where the loss curves are unknown and must be predicted online, coupling learning progress with resource allocation (CoRE-Learning).
Approach
Fits each task's loss curve with a discounted weighted least-squares power-law model updated online, converts predicted remaining data needs into an allocation problem solved exactly by a O(K) adaptive binary-tree search, and adds an explore-then-exploit phase to reduce prediction-allocation compounding errors.
Key results
- On the pure task bundle, LARA completes 4 of 5 threads while Uniform Allocation and Shortest Thread First complete 0, and LRF and ETF complete 2.
- On the mixed task bundle with epsilon set 1, LARA completes 8 of 10 threads versus 6 for Uniform Allocation and 3-4 for other baselines.
- Weighted least squares gives lower average prediction error than least squares, M4 and Engression on the pure bundle, and a single prediction round takes about 0.0007 seconds for 100 threads.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LARA instead of Uniform Allocation | CIFAR-10 pure task bundle · CIFAR-10 pure task bundle | successful thread number: 0 -> 4 | Helped |
| Use LARA instead of Least Resources First or Easiest Thread First | CIFAR-10 pure task bundle · CIFAR-10 pure task bundle | successful thread number: 2 -> 4 | Helped |
| Use LARA instead of the strongest reported baseline, Uniform Allocation | Mixed task bundle with ϵ1k · Mixed task bundle (ϵ1k) | successful thread number: 6 -> 8 | Helped |
| Use LARA instead of the strongest reported baseline, Shortest Thread First | Mixed task bundle with ϵ2k · Mixed task bundle (ϵ2k) | successful thread number: 5 -> 6 | Helped |
| Use weighted least squares instead of regular least squares for prediction | Loss-curve resource prediction · Pure task bundle and mixed task bundle | average prediction error | Helped |
| Use weighted least squares instead of M4 or Engression for prediction | Loss-curve resource prediction · Pure task bundle | average prediction error | Helped |
| Use weighted least squares instead of Engression for prediction | Loss-curve resource prediction · Mixed task bundle | average prediction error | Hurt |
| Use weighted least squares instead of M4 or Engression for prediction | Loss-curve resource prediction · Single-round prediction | average time (seconds): 9.73 -> 4.13e-05; 2.68 -> 4.13e-05 | Helped |
| Vary the exploration threshold Hk; avoid values that are too small or too large (ablation) | LARA · Mixed task bundle (ϵ1k, ϵ2k) | successful thread number: 4000 − 6000 | Mixed |
| Vary the discounted factor γ; use an intermediate region rather than values near 0.5 or 1 (ablation) | LARA weighted least squares estimator · Mixed task bundle (ϵ1k, ϵ2k) | successful thread number: 0.85 − 0.95 | Mixed |
Surprising findings
- UCB-style optimistic exploration did not outperform the simple explore-then-exploit strategy in preliminary experiments.
- Simple weighted least squares outperformed dedicated extrapolation methods (M4, Engression) on prediction error for the pure task bundle.
Limitations
- Relies on a predefined exploration threshold H_k that requires empirical tuning.
- Uses weighted least squares, an interpolation-style estimator, for extrapolation; no theoretical generalization guarantee for the overall process.
- Experiments use small models and short time units (seconds) on a single shared setting with synthetic deadlines and thresholds.
Experimental setup
Datasets: CIFAR-10 IMDB Montezuma's Revenge Yesno MNIST
Benchmarks: CIFAR-10 pure task bundle mixed task bundle (CIFAR-10, IMDB, Montezuma's Revenge, Yesno) MNIST scalability task bundles
Metrics: number of successful threads average prediction error average time per prediction/allocation round
Baselines: Uniform Allocation Shortest Thread First Least Resources First Easiest Thread First Least Squares M4 estimator Engression
Builds on: CoRE-Learning explore-then-exploit power-law loss scaling
Closest papers by meaning
- ATA: Adaptive Task Allocation for Efficient Resource Management in Distributed Machine Learning (ICML 2025 · similarity 0.87)
- Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws (ICLR 2025 · similarity 0.84)
- Algorithms with Calibrated Machine Learning Predictions (ICML 2025 · similarity 0.83)
- RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget (NeurIPS 2025 · similarity 0.83)
- Learning-Augmented Algorithms with Explicit Predictors (NeurIPS 2024 · similarity 0.83)
- Leaving the Nest: Going beyond Local Loss Functions for Predict-Then-Optimize (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.