Research map / Reinforcement learning / Visual and representation RL

A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a methodology with a hyperparameter sensitivity metric, a performance-sensitivity plane, and effective hyperparameter dimensionality, and applies it to PPO normalization variants on Brax MuJoCo tasks.

Problem

Existing RL evaluation focuses on benchmark performance and lacks a scalable, agreed-upon way to quantify how much an algorithm's peak performance depends on per-environment hyperparameter tuning.

Approach

Defines hyperparameter sensitivity as the gap between per-environment tuned normalized score and the best fixed (cross-environment) hyperparameter setting's score, plus a performance-sensitivity plane for comparing variants and an effective hyperparameter dimensionality metric counting how many hyperparameters must be tuned per environment to reach 95% of peak performance.

Key results

• Advantage per-minibatch zero-mean normalization increased performance more than sensitivity (Region 2), while advantage percentile scaling variants increased sensitivity more than performance (Region 4).
• No normalization variant improved both performance and reduced hyperparameter sensitivity relative to unnormalized PPO.
• Value target symlog lowered performance and may slightly increased sensitivity (Region 5); observation normalization may slightly reduce sensitivity but confidence intervals are wide.

Measured effects reported in the paper

Surprising findings

• Normalization variants that increased tuned performance also increased hyperparameter sensitivity, contrary to the view that normalization makes RL easier to tune.
• Algorithms with similar sensitivity can have very different effective hyperparameter dimensionality.

Limitations

• Sensitivity metric depends heavily on the chosen environment distribution; findings limited to the five Brax MuJoCo environments.
• Results depend on grid granularity and the choice of score normalization method (5th-95th percentile).
• Only PPO and its normalization variants were studied; AutoRL methods and other algorithms not evaluated.

Experimental setup

Datasets: Brax Ant Brax Halfcheetah Brax Hopper Brax Swimmer Brax Walker2d

Benchmarks: Brax Ant Brax Halfcheetah Brax Hopper Brax Swimmer Brax Walker2d

Metrics: area under learning curve normalized environment score hyperparameter sensitivity per-environment tuned score effective hyperparameter dimensionality

Baselines: PPO (unnormalized) advantage per-minibatch zero-mean normalization advantage percentile scaling advantage lower bounded percentile scaling value target symlog observation zero-mean normalization observation symlog

Builds on: PPO DreamerV3 PureJaxRL

Closest papers by meaning

• Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning (ICML 2025 · similarity 0.88)
• Position: Benchmarking is Limited in Reinforcement Learning Research (ICML 2024 · similarity 0.87)
• Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data (ICML 2025 · similarity 0.86)
• Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.86)
• A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing (NeurIPS 2026 · similarity 0.86)