Research map / Architectures and efficiency / RNNs and normalization
Hadamax Encoding: Elevating Performance in Model-Free Atari
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Hadamax encoder (max-pooled Hadamard products of GELU-activated parallel conv layers) and plugs it into PQN, reporting large gains on Atari-57 over PQN and Rainbow-DQN.
Problem
Pixel-based model-free RL on Atari-57 relies on simple encoders; scaling or modifying encoder architectures has yielded small or unstable gains, so better encoders for value-based RL are needed.
Approach
A shallow 3-layer Nature-style encoder where convolutional downsampling is replaced by max-pooling, each hidden layer is a Hadamard product of two parallel LayerNorm+GELU conv branches, and the final features are max-pooled before the linear layer; no algorithmic hyperparameter changes are needed.
Key results
- Hadamax-PQN achieves roughly 80% performance gain over vanilla PQN on Atari-57 at 200M frames (described as almost two-fold).
- Hadamax-PQN surpasses Rainbow-DQN's official 200M-frame score after about 90M frames, versus about 260M for PQN.
- Adding the Hadamax encoder to C51 improves Atari-10 performance by about 70%; DQN and Rainbow improve about 20% and 30% on Atari-3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the PQN encoder with Hadamax | PQN · Atari-57 | performance: 80% | Helped |
| Replace the C51 encoder with Hadamax | C51 · Atari-10 | performance: 70% | Helped |
| Replace the DQN encoder with Hadamax | DQN · Atari-3 | performance: 20% | Helped |
| Replace the Rainbow encoder with Hadamax | Rainbow · Atari-3 | performance: 30% | Helped |
| Add max-pooling down-sampling (ablation) | Hadamax-PQN · Atari-57 | median human-normalized score | Helped |
| Add convolutional Hadamard representations (ablation) | Hadamax-PQN · Atari-57 | median human-normalized score | Helped |
| Use GELU instead of ReLU (ablation) | Hadamax-PQN · Atari-57 | median human-normalized score | Helped |
| Increase the Hadamax encoder depth to 5 or 7 convolutional layers (ablation) | Hadamax-PQN · Atari-57 | median human-normalized score | No clear effect |
Surprising findings
- Hadamard products are successfully applied to zero-saturating activations (ReLU/GELU) in PQN, contrary to prior DQN instability reports.
- Simply deepening the Hadamax encoder with extra convolutional layers did not improve performance.
Limitations
- Not tested on more complex algorithms such as BBF on Atari-100k or model-based methods such as Dreamer.
- Extra computational overhead relative to the Nature CNN encoder used by PQN.
- Ablations use 40M frames rather than the full 200M budget.
Experimental setup
Datasets: Atari-57 Atari-10 Atari-3
Benchmarks: Atari-57 Atari-10 Atari-3
Metrics: median human-normalized score Atari-57 score profile per-game score effective rank fraction of dead neurons
Baselines: DQN C51 Rainbow-DQN PQN PQN (ResNet-15) DDQN
Builds on: PQN Hadamard representations Impala ResNet-15 Rainbow-DQN cleanrl
Closest papers by meaning
- Hadamard Representation: Scaffolding Performance Across Model-free RL (NeurIPS 2026 · similarity 0.84)
- Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC (ICML 2025 · similarity 0.82)
- Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning (ICLR 2025 · similarity 0.81)
- Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder (AAAI 2025 · similarity 0.81)
- HOT: Hadamard-based Optimized Training (CVPR 2025 · similarity 0.80)
- Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.