Research map / Probabilistic and scientific modeling / Molecules and proteins

Instance-Conditional Timescales of Decay for Non-Stationary Learning

AAAI 2024 · Probabilistic and scientific modeling › Molecules and proteins · approach: Prediction-powered and conformal inference · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MuScaTeL, which learns an instance-conditional scorer over a mixture of exponential time-decay bases, trained via a nested bilevel objective to maximize forward transfer under slow concept drift.

Problem

Slow concept drift in batch and continual learning: naive recency weighting or exponential downweighting of past data loses useful information, while equal weighting overemphasizes stale data.

Approach

Model instance importance as a learned, instance-conditional linear mixture of exponential time-decay bases (g(x)^T exp(-a(T-t))), with the scorer network trained by a bilevel objective that minimizes loss on a small post-training validation window so that the classifier maximizes forward transfer.

Key results

• On the GEO dataset, MuScaTeL reduces year-wise error vs ERM by 6.7% to 15% relative (year 5 error 47.2 vs ERM 41.0 accuracy-equivalent table reports gains up to 15%) and up to 8.8% over the nearest SOTA baseline.
• On Wild-Time, MuScaTeL achieves the best ID and OOD accuracy on all five datasets, e.g., Yearbook OOD 82.78 vs ERM 79.50 and MIMIC OOD 64.12 vs ERM 61.33.
• On SDRIFT regression datasets, MuScaTeL gives the lowest RMSE on all five datasets (e.g., Gas 0.38 vs 0.40 for the best prior method).

Measured effects reported in the paper

Surprising findings

• The method deliberately trades off past accuracy for recent/future accuracy, with training accuracy decreasing after about 40 epochs while test accuracy keeps rising.
• Standard continual learning methods were substantially worse than batch training on the GEO data, even with a frozen batch model outperforming CL variants for about 3.5 years.

Limitations

• Continual learning adaptation only reweights within each bucket (about 3 months) because older data is unavailable.
• The scorer is restricted to a separable instance-times-age form; more general scorers were not reported to help in the main text.
• Hessian-inverse approximation via Neumann series assumes many optimization steps; an alternating-update variant trades accuracy for runtime.

Experimental setup

Datasets: YFCC100M geolocalization (GEO) Wild-Time (Yearbook, FMoW-time, MIMIC, Huffpost, arxiv) CLEAR-100 SDRIFT regression datasets (Wind, Airline, Gas, News, Traffic)

Benchmarks: GEO geolocalization Wild-Time CLEAR-100 SDRIFT regression datasets

Metrics: classification accuracy error rate RMSE

Baselines: ERM MetaWeightNet Fast sample reweighting RHO-loss GroupDRO CORAL IRM LISA mixup Reservoir Sampling MiR GDUMB

Builds on: MetaWeightNet RHO-loss Wild-Time benchmark PoLRS GradCAM

Closest papers by meaning

• Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements (CVPR 2024 · similarity 0.85)
• Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.85)
• PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.85)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.84)
• LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.84)
• The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.84)