Research map / Reinforcement learning / Imitation learning
Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NEAR, a non-adversarial imitation-from-observation method that learns noise-conditioned energy functions via denoising score matching on expert state transitions and uses them as RL rewards with an annealing schedule over noise levels.
Problem
Imitation from observation (state-only expert trajectories) for physics-dependent humanoid motion, where adversarial imitation learning suffers from unstable training, non-smooth rewards and non-stationarity.
Approach
Learn a multi-scale energy function of the expert's state-transition distribution with denoising score matching (NCSN-style), use it as a fixed, smooth reward for PPO, and anneal the noise level up or down based on the policy's imitation progress so the reward stays well-defined on policy-generated samples.
Key results
- NEAR and AMP are roughly similar on the main metrics; NEAR has lower (closer-to-expert) spectral arc length on walking (-7.52 vs -8.78 for AMP) while AMP has comparable pose error (0.51 vs 0.51 on walking).
- NEAR outperforms AMP on the target-reaching (walking) composed task with pose error 0.94 vs 1.09 and task return 2.75 vs 2.23.
- NEAR performs poorly in single-clip imitation (e.g., mummy walk pose error 0.66 vs AMP 0.41), and annealing improved crane pose pose error from 1.38 to 0.94.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the unconditioned task reward to the energy reward (ablation) | NEAR with fixed σ5 · Walking and Running | average pose error: 0.49 ± 0.22 -> 0.42 ± 0.02; 0.62 ± 0.18 -> 0.57 ± 0.04 | Helped |
| add the unconditioned task reward to the energy reward (ablation) | NEAR with fixed σ5 · Crane Pose | average pose error: 1.38 ± 0.8 -> 1.23 ± 0.07 | Helped |
| use annealing instead of fixed σ5 (ablation) | NEAR energy reward · Walking and Running | average pose error: 0.49 ± 0.22 -> 0.51 ± 0.15; 0.62 ± 0.18 -> 0.62 ± 0.17 | No clear effect |
| use annealing instead of fixed σ5 (ablation) | NEAR energy reward · Crane Pose | average pose error: 1.38 ± 0.8 -> 0.94 ± 0.15 | Helped |
| replace annealing with a fixed noise level σ5 (ablation) | NEAR energy reward · Crane Pose | average pose error: 0.94 ± 0.15 -> 1.38 ± 0.8 | Hurt |
| use NEAR instead of AMP | imitation policy · Running | average pose error: 0.65 ± 0.01 -> 0.62 ± 0.17 | Helped |
| use NEAR instead of AMP | imitation policy · Walking | average pose error: 0.51 ± 0.07 -> 0.51 ± 0.15 | No clear effect |
| use NEAR instead of AMP | imitation policy · Left Punch, Crane Pose, Mummy Walk, and Spin Kick | average pose error: Left Punch: 0.32 ± 0.01 -> 0.37 ± 0.05; Crane Pose: 0.82 ± 0.09 -> 0.94 ± 0.15; Mummy Walk: 0.41 ± 0.01 -> 0.66 ± 0.39; | Hurt |
Surprising findings
- Annealing at higher noise levels causes drops in energy reward return, introducing non-stationarity.
- NEAR performs poorly in single-clip imitation tasks, whereas AMP is less affected by data scarcity.
Limitations
- Sensitivity to the noise scale; degradation when annealing at higher noise levels.
- Poor performance in single-clip, data-limited imitation settings.
- Policies converge to locally optimal behaviour when the dataset contains bi-directional state transitions.
Experimental setup
Datasets: CMU motion capture SFU motion capture
Benchmarks: CMU/SFU humanoid imitation tasks (walking, running, left punch, crane pose, mummy walk, spin kick) target reaching and punching composed tasks
Metrics: average dynamic time warping pose error spectral arc length (SPARC) task return root body velocity root body jerk wall-clock time
Baselines: AMP
Builds on: Adversarial Motion Priors Noise Conditioned Score Networks Proximal Policy Optimisation Generative Adversarial Networks
Closest papers by meaning
- Recovering Hidden Reward in Diffusion-Based Policies (ICML 2026 · similarity 0.89)
- Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective (AAAI 2024 · similarity 0.88)
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.87)
- Imitation from Observations with Trajectory-Level Generative Embeddings (NeurIPS 2026 · similarity 0.87)
- Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation (NeurIPS 2024 · similarity 0.87)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.