Research map / Label-efficient and robust learning / Recognition and long-tail
RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RESQUE, a forward-pass-only index (cosine-angle shift for distribution shift; complement of Adjusted Rand Index on clustered representations for task change) that predicts the retraining cost of a model in epochs, gradient norm, parameter change, energy and carbon.
Problem
Reusing a deep model by retraining it is more sustainable than training from scratch, but there is no cheap way to estimate the retraining cost of a model before retraining it under distribution shift or a change of task.
Approach
Quantify retraining cost from representation geometry computed with one forward pass: for distribution shift, average the arccos angle between class-wise normalized summed embeddings of original and shifted data; for task change, retrain for one epoch on the new task, cluster the representations, and take the complement of the Adjusted Rand Index between clusters and true labels.
Key results
- ResNet18 on CIFAR100 under noise: Spearman 0.97 between RESQUE and epochs (p=1e-19), with GradNorm 0.97 and parameter change 0.95.
- ViT on CIFAR10 under noise: Pearson 0.87 with epochs (p=2e-10) and Spearman 0.88 across measures.
- For task change, ResNet18 on CIFAR10 target reports Pearson 0.86 with epochs (p=0.012) and Spearman 0.96 (p=4e-04) across original-task models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RESQUE_dist as an estimate of retraining cost under distribution shifts | ResNet18 · CIFAR100 distribution shifts | Spearman correlation with epochs: 0.97 (1e-19) | Helped |
| Use RESQUE_dist as an estimate of retraining cost under distribution shifts | ViT · CIFAR10 distribution shifts | Pearson correlation with GradNorm: 0.88 (1e-10) | Helped |
| Use RESQUE_dist as an estimate of retraining cost under distribution shifts | VGG16 · SVHN distribution shifts | Pearson correlation with GradNorm: 0.95 (1e-16) | Helped |
| Use RESQUE_task to estimate retraining cost for new tasks | ResNet18 · CIFAR10 target task | Spearman correlation with epochs: 0.96 (4e-04) | Helped |
| Use RESQUE_task to estimate retraining cost for new tasks | ViT B/16 · CIFAR10 target task | Pearson correlation with parameter change: 0.92 (0.001) | Helped |
| Initialize KMeans with original-data centroids instead of Kmeans++ or random assignment (ablation) | RESQUE_task clustering · New target tasks | similarity of clustering labels | No clear effect |
| Retrain all model layers instead of only the final few layers (ablation) | New-task model adaptation · New target tasks | accuracy | Helped |
Surprising findings
- Retraining to a new task initially shows higher gradient norm and parameter change than retraining to distribution shift, because the new task's features are unknown to the model.
Limitations
- Some prior-task CNN models could not reach high accuracy on STL10 and were excluded from task-change experiments
- Correlations on small sets of target tasks and models with p-values sometimes above 0.05 (e.g., ResNet18 on GTSRB parameter change, CIFAR100 task correlations)
- Energy and carbon measurements taken on a single RTX 2060 node
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN Food101 EMNIST Fashion MNIST GTSRB MNIST STL10
Benchmarks: CIFAR10 CIFAR100 SVHN GTSRB Food101 EMNIST Fashion MNIST MNIST
Metrics: Pearson correlation Spearman correlation p-value epochs total gradient norm normalized parameter change energy consumption carbon emissions
Builds on: Adjusted Rand Index KMeans++
Closest papers by meaning
- When to retrain a machine learning model (ICML 2025 · similarity 0.87)
- Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning (AAAI 2024 · similarity 0.85)
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.84)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.84)
- You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.84)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.