Research map / Reinforcement learning / Visual and representation RL
Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Induces functional modules in RL policy networks via spatially aware (distance-weighted, log-sparsity) regularisation and neuron relocation, detects them with an extended Louvain method using a correlation-alignment (ARI) metric, and characterises them by weight interventions in MiniGrid and Pong.
Problem
Interpretability of RL policies is hard to scale when done at the level of neurons or decision nodes; standard community detection (Louvain) also fails when applied to MLP weight graphs because of layer-wise connectivity and input fan-out.
Approach
Penalise length-weighted log sparsity of weights in a 2D-embedded MLP (with periodic neuron relocation) to induce modules, then detect them with an extended Louvain that adds an isolation metric and an ARI-based alignment between weight-based and activation-correlation partitions, and validate modules by negative saturation/negation of module weights.
Key results
- Fine-tuning raises average module isolation by 16.8% and correlation alignment by 33.7% across regularisation strengths; the fine-tuned internal Louvain adds a further 2.4% isolation.
- At module emergence, returns drop by about 11.4% (DO) and 12.5% (3D-DO) versus 0.8% for G2K, with 90.5%, 96.5% and 89% fewer parameters respectively.
- Negatively saturating a module reduces forward/backward actions by 83% in one 3D-DO network; negation of a module reduces return by 73% versus 36% for saturation in the 3D-DO network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatially aware regularisation | PPO policies · DO, 3D-DO, G2K at module emergence | return: 11.4% (DO), 12.5% (3D-DO), and 0.8% (G2K) decrease | Hurt |
| Add spatially aware regularisation | PPO policies · DO, 3D-DO, G2K at module emergence | parameter count: 90.5%, 96.5%, and 89% fewer parameters | Helped |
| Add regularisation and neuron relocation | PPO policies · DO, 3D-DO, G2K | training time: 17% | Hurt |
| Add fine-tuning to the initial Louvain partition | Louvain module detection · DO, 3D-DO, G2K | module isolation: 16.8% | Helped |
| Add fine-tuning to the initial Louvain partition | Louvain module detection · DO, 3D-DO, G2K | correlation alignment: 33.7% | Helped |
| Initialize fine-tuning with internal Louvain | Fine-tuned Louvain detection · DO, 3D-DO, G2K | module isolation: 2.4% | Helped |
| Initialize fine-tuning with internal Louvain | Fine-tuned Louvain detection · DO, 3D-DO, G2K | correlation alignment: -1.6% | Hurt |
| Remove distance weighting from the connection-cost regulariser (ablation) | Regularised PPO policies · DO, 3D-DO, G2K | module isolation | Hurt |
| Remove neuron relocation (ablation) | Regularised PPO policies · DO, 3D-DO, G2K | module isolation | Hurt |
| Remove distance weighting from the connection-cost regulariser (ablation) | Regularised PPO policies · DO, 3D-DO, G2K | correlation alignment (ARI) | Hurt |
| Remove neuron relocation (ablation) | Regularised PPO policies · DO, 3D-DO, G2K | correlation alignment (ARI) | No clear effect |
| Use logarithmic rather than L1-based sparsity regularisation (ablation) | Regularised PPO policies · MiniGrid environments | module isolation and ARI relative to return | Helped |
Surprising findings
- Negating a module's weights hurts performance far more than saturating it, indicating perturbation is more damaging than disabling.
- In the Pong policy, the learned network relies on the opponent's position, a proxy-gaming flaw; removing it yields robust policies with only minor performance loss.
Limitations
- Strong trade-off between interpretability (regularisation strength) and return, especially in DO and 3D-DO.
- Evaluated only on small MiniGrid and Pong environments with small MLPs (two hidden layers of 32 neurons).
Experimental setup
Datasets: MiniGrid Dynamic Obstacles MiniGrid Go-to-Key MiniGrid 3D Dynamic Obstacles Pong (custom JAX implementation)
Benchmarks: MiniGrid Dynamic Obstacles MiniGrid Go-to-Key MiniGrid 3D Dynamic Obstacles Pong
Metrics: mean return / success rate module isolation adjusted Rand index (correlation alignment) parameter count action frequency under interventions
Baselines: Louvain (standard) vanilla PPO L1 regularisation
Builds on: Louvain algorithm PPO spatially embedded neural network regularisation MiniGrid
Closest papers by meaning
- Efficient Reinforcement Learning by Discovering Neural Pathways (NeurIPS 2024 · similarity 0.87)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.85)
- Neural networks are more modular than single neurons suggest (NeurIPS 2026 · similarity 0.85)
- Adaptive Rational Activations to Boost Deep Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization (ICLR 2025 · similarity 0.85)
- Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.