Research map / Reinforcement learning / Visual and representation RL

CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CrystalBox, a model-agnostic posthoc framework that explains input-driven DRL controllers by predicting decomposed future returns per reward component via supervised learning on Monte Carlo rollouts over training traces.

Problem

Feature-based explainers (e.g., LIME) cannot distinguish actions or states in input-driven DRL controllers because they ignore the controller's view of future consequences, and existing future-based explainers need agent modifications or accurate environment models that are unavailable in input-driv

Approach

Decompose the Q-function into per-reward-component returns and learn a supervised predictor of these decomposed future returns from unbiased Monte Carlo rollouts over logged input traces, so that no environment model or controller modification is needed; the predictor gives sub-10ms explanations for discrete and continuous control.

Key results

• CrystalBox achieves the lowest squared error to ground-truth decomposed returns in all three return components for both ABR and CC, under factual and counterfactual actions, outperforming naive and distribution-aware sampling.
• CrystalBox inference latency is under 10 ms, versus 50-250 ms for sampling-based methods.
• For large performance-drop event detection, CrystalBox shows high recall with low false-positive rates on factual actions in ABR and CC.

Measured effects reported in the paper

Surprising findings

• Jointly training the controller and the explainer with a weighted auxiliary loss degrades policy quality when weighting the explainer loss more, and the explainer does not improve, so joint training is non-trivial.

Limitations

• High variance of Monte Carlo return samples due to dependence on input-trace transitions requires many samples.
• CC false-positive rates are high for high-loss events.
• Joint training of controller and explainer degrades performance and does not improve the explainer.

Experimental setup

Datasets: Puffer live-streaming traces synthetic congestion control traces

Benchmarks: Puffer ABR traces Congestion control traces

Metrics: L2 fidelity error recall false positive rate inference latency

Baselines: LIME Naive sampling Distribution-Aware sampling

Builds on: LIME Monte Carlo Policy Evaluation Input-Driven MDP Aurora

Closest papers by meaning

• RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation (ICML 2024 · similarity 0.87)
• ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies (NeurIPS 2026 · similarity 0.85)
• Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes (AAAI 2024 · similarity 0.85)
• Reward Machines for Deep RL in Noisy and Uncertain Environments (NeurIPS 2024 · similarity 0.84)
• Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning (AAAI 2024 · similarity 0.84)
• SMoSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks (AAAI 2025 · similarity 0.84)