Research map / Label-efficient and robust learning / Recognition and long-tail
Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Feature Dimensionality Reduction network plus an online VQ-VAE-style clustering module that plugs into PPO, yielding stable, automatically discovered semantic clusters of Procgen states for interpretability analysis.
Problem
Interpreting DRL feature spaces: prior semantic clustering analyses rely on t-SNE, which is unstable across seeds and sample sizes, and require heavy manual annotation of states.
Approach
Train a 2D Feature Dimensionality Reduction net jointly with PPO using a Student-t pairwise-similarity loss, and cluster the FDR features online with a modified VQ-VAE codebook (second term only, acting as online k-means); the discrete cluster code is added back to the state feature so the policy is conditioned on it.
Key results
- Human evaluators (15) rated cluster consistency and interpretability mostly above 4 on a 5-point Likert scale across Jumper, FruitBot and Ninja.
- Across nine runs, the learned 2D FDR space achieves the best silhouette, Davies-Bouldin and Calinski-Harabasz scores versus UMAP, Isomap, PCA, MDS and t-SNE, with CH an order of magnitude larger than baselines.
- The module has minimal impact on PPO performance on Procgen easy-mode levels, and the stop-gradient ablation still shows clustering but with fuzzier boundaries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · FruitBot | cluster transition probability: 0.1520 -> 0.1081 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Jumper | cluster transition probability: 0.3015 -> 0.2224 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Ninja | cluster transition probability: 0.2705 -> 0.2680 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · FruitBot | ECE: 1.9068 -> 1.8886 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · FruitBot | TCA@3: 0.7050 -> 0.7302 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · FruitBot | TCA@6: 0.5309 -> 0.5502 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Jumper | ECE: 0.6987 -> 0.5572 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Jumper | TCA@3: 0.7040 -> 0.7911 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Jumper | TCA@6: 0.6715 -> 0.7089 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Ninja | ECE: 1.5505 -> 1.3086 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Ninja | TCA@3: 0.5798 -> 0.6787 | Helped |
| add the semantic clustering module rather than stopping its gradients (ablation) | PPO · Ninja | TCA@6: 0.3923 -> 0.4985 | Helped |
Surprising findings
- Semantic clusters persist even when the clustering module's gradients are stopped, suggesting clustering is an intrinsic property of DRL feature spaces.
- The Jumper agent's cluster assignment depends on the radar and distance bar rather than the presence of carrots, which the authors flag as a risk when sensor and visual inputs disagree.
Limitations
- Relies on clear semantic distributions; clusters become ambiguous when policies are far from optimal.
- Requires manually selecting the number of clusters (8 used), with too many fragmenting semantics and too few merging them.
- Policy interpretations of clusters are written manually rather than automatically summarized.
Experimental setup
Datasets: Procgen (Ninja, Jumper, FruitBot, CoinRun)
Benchmarks: Procgen (Ninja, Jumper, FruitBot)
Metrics: mean score silhouette score Davies-Bouldin index Calinski-Harabasz score episode cluster entropy temporal cluster agreement cluster transition probability human Likert ratings
Baselines: PPO t-SNE UMAP Isomap PCA MDS Stop-Grad ablation
Builds on: PPO IMPALA VQ-VAE t-SNE Procgen
Closest papers by meaning
- Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning (ICML 2025 · similarity 0.85)
- LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning (ICLR 2025 · similarity 0.84)
- VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning (CVPR 2025 · similarity 0.84)
- Mixtures of Experts Unlock Parameter Scaling for Deep RL (ICML 2024 · similarity 0.84)
- Meta-Reinforcement Learning via Exploratory Task Clustering (AAAI 2024 · similarity 0.84)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.