Research map / Reinforcement learning / RL from feedback

Aligning Transformers with Continuous Feedback via Energy Rank Alignment

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes energy rank alignment (ERA), a gradient-based, RL-free objective that fine-tunes autoregressive policies (molecular transformers and ESM3-1.4B) toward a Gibbs-Boltzmann distribution defined by an explicit reward/energy function.

Problem

Aligning autoregressive generative models (chemical language models and protein language models) to externally specified, easily evaluable properties when human preference data is unavailable or costly, and existing DPO/PPO objectives are either greedy or lack a clean regularization/entropy control.

Approach

Minimize a KL divergence between a regularized Boltzmann preference probability (computed directly from the explicit energy and the reference policy) and the policy's Bradley-Terry preference probability; the minimizer is a Gibbs-Boltzmann distribution with inverse temperature beta and KL-regularization gamma controlled independently, and it relates to PPO and DPO objectives.

Key results

• On JNK3 top-100, ERA achieves mean score 0.987 ± 0.001 with IntDiv 0.264, exceeding MolRL-MGPT (0.961) and other baselines.
• On GSK3beta top-100, ERA reaches mean score 0.996 with IntDiv 0.219, more diverse than MolRL-MGPT (0.362 IntDiv) but marginally below its 1.000 mean score.
• Aligning ESM3-1.4B on TrpB with higher beta samples mutants reaching the highest possible EVmutation score in a single alignment round.

Measured effects reported in the paper

Surprising findings

• DPO's chemical validity collapsed to 0% over later checkpoints in an online QED experiment, while ERA degraded only slightly.
• Without KL regularization (gamma=0), prompted molecular alignment produced under 25% chemically valid molecules.

Limitations

• Requires an explicit, reasonably cheap reward model, which is hard to design for complex tasks like binding affinity.
• Does not train the molecular generator for synthetic accessibility, so generated molecules may be hard to synthesize.
• Theoretical convergence of gradient descent on the parametric policy to the global minimizer is not proven.

Experimental setup

Datasets: ChEMBL TrpB EVmutation oracle sequences

Benchmarks: GSK3beta docking oracle top-100 JNK3 docking oracle top-100 RDKit property oracles (QED, LogP, Ring Count, MR) EVmutation score for TrpB

Metrics: mean score internal diversity chemical validity top-10 AUC mean property value

Baselines: DPO MolRL-MGPT GFlowNet GraphGA JT-VAE REINVENT PPO PPOD

Builds on: direct preference optimization proximal policy optimization Bradley-Terry model

Closest papers by meaning

• Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization (NeurIPS 2024 · similarity 0.89)
• Searching for High-Value Molecules Using Reinforcement Learning and Transformers (ICLR 2024 · similarity 0.89)
• Efficient Evolutionary Search Over Chemical Space with Large Language Models (ICLR 2025 · similarity 0.88)
• ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design (NeurIPS 2026 · similarity 0.87)
• Amortized Molecular Optimization via Group Relative Policy Optimization (NeurIPS 2026 · similarity 0.87)
• Domain-Agnostic Molecular Generation with Chemical Feedback (ICLR 2024 · similarity 0.87)