Research map / Large language models / Language, safety and interpretability

Embedding-Aligned Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EAGLE, an RL agent that treats a pre-trained LLM as an environment and steers generated movie descriptions toward high-utility regions of a behavioral embedding space, using a G-optimal design action set.

Problem

Aligning LLM text generation with objectives defined in a latent embedding space (e.g., surfacing content gaps in recommender systems), where decoding optimal embeddings back to valid entities is unreliable.

Approach

Instead of training an ELM decoder over the embedding space, an RL agent iteratively issues action prompts to a fixed environment LLM, encodes each new entity with an encoder into the embedding space, and is rewarded by a utility; the action set is state-dependent and shaped by LLM-generated candidates and a G-optimal design reference policy.

Key results

• EAGLE with G-optimal design reaches utility 0.74 ± 0.03 and 76% user-utility preference from 124 human raters, versus 31% for the ELM baseline.
• Environment transfer: an EAGLE agent trained with Gemini Pro and tested on GPT-4 reaches 86% user utility and 78% rater utility.
• Removing personalized actions drops utility from 0.75 to 0.56, and macro-action space degrades performance relative to default actions.

Measured effects reported in the paper

Surprising findings

• ELM decoders underperform EAGLE on user and rater utility, though raters judge ELM outputs as more distant from the anchor.
• Training EAGLE with the stronger Gemini Ultra environment did not improve results over Gemini Pro.

Limitations

• Coverage is limited by the textual action space; some embedding directions may be unreachable.
• Performance degrades on highly rated movies, where the MovieLens rating scale caps utility.
• Training is expensive because each step queries an environment LLM.

Experimental setup

Datasets: MovieLens 25M Amazon Review

Benchmarks: MovieLens 25M content gap utility

Metrics: utility U(z) user utility (human preference rate) rater utility (human preference rate) distance score

Baselines: ELM Uniform reference policy Optimistic action reference policy

Builds on: Embedding Language Models (ELM) KL-regularized policy gradient (RLHF) G-optimal design for contextual bandits

Closest papers by meaning

• Controlling Large Language Model with Latent Action (ICML 2025 · similarity 0.88)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.87)
• EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)
• Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.87)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.87)
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.87)