Research map / Reinforcement learning / Visual and representation RL
An Information Theoretic Approach to Interaction-Grounded Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VI-IGL and f-VI-IGL, information-theoretic methods that learn a latent-reward decoder by minimizing conditional mutual information between context-action and feedback (with a regularizer), solved via a variational min-max formulation, and shows gains over E2G on noisy MNIST number-guessing.
Problem
Interaction-grounded learning requires inferring a latent binary reward from feedback; existing IGL methods rely on full conditional independence of feedback and context-action given reward, which breaks down under noisy feedback.
Approach
Learn the reward decoder by minimizing conditional MI I(Y;X,A|R) regularized by -beta*I(X,A;R), and estimate both MI terms with Donsker-Varadhan / f-divergence variational representations, yielding a min-max problem solved by gradient descent-ascent.
Key results
- In the noiseless setting VI-IGL (beta=10) achieves 81.6±7.9% vs E2G 82.2±4.3%, comparable.
- At noise level 0.1 with independent noise, VI-IGL (beta=10) achieves 71.0±16.2% vs E2G 25.0±18.4%.
- Regularization matters: at noise 0.1 VI-IGL beta=0 gets 52.4±25.3 to 65.2 range vs beta=10 improving e.g. context-inclusive from 52.4% to 72.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VI-IGL with β=10 instead of E2G | Number-guessing task · Noisy feedback, noise levels 0.1, 0.2, and 0.3 | accuracy | Helped |
| use VI-IGL with β=10 instead of E2G | Number-guessing task · Noiseless | accuracy: 81.6±7.9 vs 82.2±4.3 | No clear effect |
| add the regularization term by using β=10 rather than β=0 (ablation) | VI-IGL · Number-guessing, noise 0.1, independent noise | accuracy: 60.6±15.5 -> 71.0±16.2 | Helped |
| add the regularization term by using β=10 rather than β=0 (ablation) | VI-IGL · Number-guessing, noise 0.1, action-inclusive noise | accuracy: 64.8±17.9 -> 73.6±16.0 | Helped |
| add the regularization term by using β=10 rather than β=0 (ablation) | VI-IGL · Number-guessing, noise 0.1, context-inclusive noise | accuracy: 52.4±25.3 -> 72.1±10.2 | Helped |
| add the regularization term by using β=10 rather than β=0 (ablation) | VI-IGL · Number-guessing, noise 0.1, context-action-inclusive noise | accuracy: 63.4±16.4 -> 65.2±16.3 | Helped |
| add the regularization term by using β=10 rather than β=0 (ablation) | VI-IGL · Number-guessing, noiseless | accuracy: 55.7±27.3 -> 81.6±7.9 | Helped |
| replace KL-KL with χ²-χ² divergences (ablation) | VI-IGL (β=10) · Number-guessing, noise types I, A, C, C-A, and noiseless | accuracy: I: 71.0±16.2 -> 72.7±17.4; A: 73.6±16.0 -> 65.3±11.1; C: 72.1±10.2 -> 76.2±11.5; C-A: 65.2±16.3 -> 70.4±15.5; N: 81.6±7. | Mixed |
| replace KL-KL with χ²-KL divergences (ablation) | VI-IGL (β=10) · Number-guessing, noise types I, A, C, C-A, and noiseless | accuracy: I: 71.0±16.2 -> 74.1±12.7; A: 73.6±16.0 -> 71.5±16.7; C: 72.1±10.2 -> 69.4±11.5; C-A: 65.2±16.3 -> 64.6±14.7; N: 81.6±7. | Mixed |
| use a context-action-feedback reward decoder instead of a feedback-only decoder (ablation) | VI-IGL (β=10, KL-KL) · Number-guessing, noise types I, A, C, C-A, and noiseless | accuracy: I: 71.0±16.2 -> 42.0±24.1; A: 73.6±16.0 -> 49.6±24.4; C: 69.3±10.9 -> 46.2±16.9; C-A: 72.1±10.2 -> 59.4±17.4; N: 81.6±7. | Hurt |
Surprising findings
- Unregularized objective (beta=0) can be highly unstable, with accuracy as low as 33.6% under action-inclusive noise at level 0.3.
- Using a context-action-feedback reward decoder performs much worse than a feedback-only decoder (e.g., 42.0% vs 71.0% under independent noise).
Limitations
- Variance of performance is large across trials, attributed to joint training of multiple networks and initialization.
- Sample complexity bound involves covering numbers that are prohibitive for deep networks; does not account for gradient-based optimization.
- Experiments limited to a single synthetic number-guessing task with injected noise; full conditional independence assumption is violated in several noise types.
Experimental setup
Datasets: MNIST EMNIST Letter
Benchmarks: number-guessing task with noisy feedback (MNIST)
Metrics: policy accuracy
Baselines: E2G
Builds on: Interaction-Grounded Learning E2G Donsker-Varadhan representation information bottleneck
Closest papers by meaning
- Provably Efficient Interactive-Grounded Learning with Personalized Reward (NeurIPS 2024 · similarity 0.87)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.86)
- InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.86)
- Information-Theoretic Reward Decomposition for Generalizable RLHF (NeurIPS 2025 · similarity 0.86)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.86)
- Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.