Research map / Reinforcement learning / Visual and representation RL
Time-Varying Propensity Score to Bridge the Gap between the Past and Present
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a time-varying propensity score that reweights past data by how similarly it evolved relative to the present, applied to supervised continuous-shift image classification and RL (SAC variants) settings.
Problem
Models deployed on gradually drifting data must decide which past data to train on; fitting to all past data is biased under shift, while using only recent data is statistically inefficient.
Approach
Generalizes the two-sample standard propensity score to a time-indexed ratio p_T(x)/p_t(x), estimated by a logistic classifier over (x, t, t') triplets, so the weights reflect both similarity to the present and similar evolution in the past; it plugs into the loss as a per-sample weight with no other changes.
Key results
- On continuous supervised benchmarks, the time-varying propensity score is reported to outperform Everything, Recent, and Fine-tune across CIFAR-10 and CLEAR shift settings, averaged over 3 seeds.
- In RL, adding the score to SAC's Q-update yields higher undiscounted return than standard SAC on ROBEL D'Claw and Half-Cheetah WindVel across 10 seeds.
- The paper reports that Method 1 (exponential-family over (x,t)) and Method 2 (deviation from marginal) perform similarly, with Method 1 having a slight edge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add time-varying propensity weights to Everything | Time-varying propensity score · Drifting Gaussian Data | accuracy | Helped |
| Add time-varying propensity weights to Everything | Time-varying propensity score · CIFAR-10a, CIFAR-10b, and CLEAR | accuracy | Helped |
| Add time-varying propensity weights to SAC | SAC · ROBEL D’Claw | undiscounted return | Helped |
| Add time-varying propensity weights to the baseline RL methods | ROBEL D’Claw and Half-Cheetah WindVel · Reinforcement learning tasks | undiscounted return | Helped |
| Use time-varying propensity scoring instead of standard propensity scoring | Continuous CIFAR-10b · CIFAR-10b | accuracy | Helped |
| Use Method 1 instead of Method 2 to estimate the time-varying propensity score (ablation) | Time-varying propensity score · Continuous CIFAR-10 and CLEAR | accuracy | No clear effect |
| Add time-varying propensity weights when there is no data shift (ablation) | CIFAR-10 · CIFAR-10 without shift | accuracy | No clear effect |
| Vary the propensity estimator training sample size (ablation) | Time-varying propensity score · CIFAR-10 at sample sizes 50, 100, and 300 | accuracy | Helped |
| Increase the number of random seeds from 3 to 8 (ablation) | Continuous supervised learning experiments · CIFAR-10a, CIFAR-10b, and CLEAR | accuracy | No clear effect |
Surprising findings
- Everything performs poorly on the Gaussian drift experiment but better than Fine-tune and Recent on continuous CIFAR-10, with no consistent winner among baselines.
- Treating (x,t) as a single standard propensity variable is reported to be not useful for the time-conditioned objective.
Limitations
- Assumes gradual drift with bounded total variation between consecutive time steps; abrupt changes are out of scope.
- Requires a fixed number of samples per time step and retraining from scratch at each step in the experiments, so no continual updating is evaluated.
- Code is promised only upon publication.
Experimental setup
Datasets: CIFAR-10 CLEAR ROBEL D'Claw MuJoCo Half-Cheetah WindVel synthetic drifting Gaussian
Benchmarks: CIFAR-10a CIFAR-10b CIFAR-10c CLEAR Gaussian drift ROBEL D'Claw Half-Cheetah WindVel
Metrics: test accuracy average undiscounted return
Baselines: Everything Recent Fine-tune standard propensity score SAC SAC + Context Meta-Weight-Net
Builds on: standard propensity score (density ratio estimation) Soft-Actor-Critic Meta-Weight-Net
Closest papers by meaning
- When to retrain a machine learning model (ICML 2025 · similarity 0.83)
- Model Assessment and Selection under Temporal Distribution Shift (ICML 2024 · similarity 0.83)
- Off-policy estimation with adaptively collected data: the power of online learning (NeurIPS 2024 · similarity 0.82)
- CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions (NeurIPS 2024 · similarity 0.82)
- Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data (NeurIPS 2024 · similarity 0.82)
- Stabilized Neural Prediction of Potential Outcomes in Continuous Time (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.