Research map / Reinforcement learning / Policy gradient and actor-critic
A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a local, round-level TracIn-style data attribution framework for PPO in online RL and uses it for diagnosis, temporal analysis, and an iterative influence-based filtering algorithm (IIF) that improves sample efficiency and runtime.
Problem
Data attribution methods assume fixed datasets, but in online RL each collected experience both updates the policy and shapes future data collection, so existing attribution does not apply; online RL is also sample-inefficient, unstable, and hard to interpret.
Approach
Attribute checkpoints to records in the current round's rollout buffer using TracIn-style gradient similarity, with two target functions (action log-probability and a reference-policy advantage-weighted return surrogate) computed at the round's starting policy, and use the resulting influence scores to filter harmful records during PPO training.
Key results
- IIF reduces training rounds needed to match standard PPO by 20-67% across environments (SE_ave), and random filtering performs worse than standard training.
- In RLHF toxicity mitigation with gpt-neo-2.7B, IIF filters about 50% of records, halves optimization time per round, and requires less than half the rounds to surpass standard training, giving about 4x total runtime reduction.
- Bottom (most harmful) records are characterized by inaccurate advantage estimates, with strongly negative Spearman correlation between influence and the product of MC and estimated advantages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter negative-influence records iteratively | PPO across standard RL environments · Standard RL environments | training rounds to match PPO performance: 20-67% reduction | Helped |
| filter negative-influence records iteratively | PPO across standard RL environments · Standard RL environments | test return | Helped |
| filter negative-influence records iteratively | RLHF PPO with GPT-Neo 2.7B · RLHF toxicity mitigation | total runtime: ∼4× reduction | Helped |
| use random filtering instead of influence-based filtering | PPO · Standard RL environments | test return | Hurt |
| use advantage-based filtering based on sign mismatch and advantage error | PPO · FrozenLake | test return | Helped |
| use advantage-based filtering based on sign mismatch and advantage error | PPO · MiniGrid | test return | Hurt |
| prioritize samples by TD error | PPO · FrozenLake | learning speed | Helped |
| prioritize samples by TD error | PPO · LunarLander | learning performance | Hurt |
| discard all negative-influence records (p=100.0%) (ablation) | IIF · Standard RL environments | final performance and learning progress | Hurt |
| filter 12.5% rather than 100.0% of negative-influence records (ablation) | IIF · LunarLander | SE_ave: 8.9\% { $\pm$ 2.0\%} -> \textbf{28.2}\% { $\pm$ 1.3\%} | Helped |
| filter 100.0% rather than 12.5% of negative-influence records (ablation) | IIF · MiniGrid | SE_ave: \textit{67.5}\% { $\pm$ 5.1\%} -> \textbf{75.4}\% { $\pm$ 3.6\%} | Helped |
| filter 50.0% rather than 12.5% of negative-influence records (ablation) | IIF · FrozenLake | SE_ave: 23.5\% { $\pm$ 3.1\%} -> \textbf{33.7}\% { $\pm$ 3.4\%} | Helped |
Surprising findings
- Discarding all negative-influence records (p=100%) is often suboptimal, attributed to non-additivity of sample influence.
- Simple advantage-based and TD-error-based filtering heuristics fail or degrade performance in more complex environments such as MiniGrid and LunarLander, while influence-based filtering still works.
Limitations
- Uses SGD as a proxy for Adam, since TracIn is designed for SGD, while the RL and LLM settings commonly use adaptive optimizers.
- The framework lacks a clear counterfactual interpretation due to circular data-model dependency.
- Monte Carlo advantage estimates needed for heuristics are only feasible in small discrete environments, and advantage estimates are noisy in larger ones.
Experimental setup
Datasets: real-toxicity-prompts Wiki-Toxic
Benchmarks: FrozenLake MiniGrid-Empty-8x8 Acrobot Highway LunarLander BipedalWalker Pong Wiki-Toxic
Metrics: test return SE_ave (sample efficiency reduction in rounds) SE_peak runtime reduction mean toxicity probability
Baselines: standard PPO random filtering advantage-based heuristic TD error based heuristic (PER-inspired)
Builds on: PPO TracIn Prioritized Experience Replay TRL
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.90)
- Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption (NeurIPS 2025 · similarity 0.89)
- Harnessing Density Ratios for Online Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.