Research map / Reinforcement learning / Visual and representation RL
LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LICORICE, a label-efficient training scheme for concept-bottleneck RL policies that interleaves concept and policy learning, uses ensemble disagreement active learning and data decorrelation to reach near-full reward with at most a few thousand concept labels, and tests GPT-4o as an automated concept annotator.
Problem
Concept-based interpretable RL requires concept labels for every training state, which is costly from humans or VLM APIs; the paper targets learning concept-bottleneck policies under a small concept-annotation budget.
Approach
Combine iterative training (alternately fit concept layer on recent on-policy data and train the action head with frozen concepts), disagreement-based active selection via a concept ensemble (query-by-committee / variance), and random acceptance-based decorrelation of rollout states, so PPO-based concept policies need only hundreds to a few thousand labels.
Key results
- LICORICE reaches 100% of max reward on PixelCartPole with 500 labels, versus 36% for the second-best budget-constrained baseline.
- LICORICE attains 99% reward on DoorKey (300 labels) and DynamicObstacles, matching the unconstrained CPM (which uses about 1M-15M labels) on those environments.
- With GPT-4o as annotator, LICORICE reaches 83% and 88% of max reward on DoorKey and DynamicObstacles, but fails on PixelCartPole, Boxing, and Pong (under 25% reward).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add iterative training across multiple policy/data iterations (ablation) | LICORICE · PixelCartPole | reward: 0.533896 -> 1.0 | Helped |
| add iterative training across multiple policy/data iterations (ablation) | LICORICE · DoorKey | concept error: 0.089535 -> 0.047504 | Helped |
| add data decorrelation (ablation) | LICORICE · DoorKey | reward: 0.783113 -> 0.990297 | Helped |
| add disagreement-based active learning to select queried samples (ablation) | LICORICE · DynamicObstacles | reward: 0.583053 -> 0.990607 | Helped |
| use LICORICE rather than budget-constrained baselines | PixelCartPole · PixelCartPole | reward: 0.363624 -> 1.0 | Helped |
| use LICORICE rather than budget-constrained baselines | DoorKey · DoorKey | reward: 0.886567 -> 0.990297 | Helped |
| use LICORICE rather than budget-constrained baselines | DynamicObstacles · DynamicObstacles | reward: 0.99253 vs 0.990607 | No clear effect |
| use GPT-4o labels with LICORICE instead of perfect concept labels | LICORICE · Boxing | reward: 1.004997 -> 0.221577 | Hurt |
| use GPT-4o labels with LICORICE instead of perfect concept labels | LICORICE · DoorKey | reward: 0.990297 -> 0.830113 | Hurt |
Surprising findings
- GPT-4o labels are effective only for simpler environments; in Boxing and Pong, LICORICE+GPT-4o achieves very low reward.
- More concept labels from the VLM do not always improve reward or concept error, likely because of added label noise.
Limitations
- VLM annotation (GPT-4o) is inaccurate for continuous or many-valued concepts, so it fails in complex environments.
- Evaluated only on five environments with image input and pre-specified concept sets; concept design is assumed.
Experimental setup
Datasets: PixelCartPole DoorKey DynamicObstacles Boxing (OCAtari) Pong (VIPER concepts)
Benchmarks: PixelCartPole DoorKey DynamicObstacles Boxing Pong
Metrics: normalized reward (fraction of PPO upper bound) concept error (MSE or 1 - accuracy)
Baselines: Sequential-Q Disagreement-Q Random-Q CPM LICORICE-IT LICORICE-DE LICORICE-AC
Builds on: PPO concept bottleneck models CPM (concept-based multi-agent RL) query-by-committee
Closest papers by meaning
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.87)
- Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents (NeurIPS 2024 · similarity 0.86)
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.86)
- Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.86)
- RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.86)
- Interpreting Emergent Planning in Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.