Research map / Large language models / Alignment and preferences

MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

AAAI 2025 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAPLE, a Bayesian active preference learning framework where LLMs model the distribution over preference functions from natural-language and pairwise trajectory feedback, with LLM-guided query selection that avoids hard-to-answer queries.

Problem

Preference learning from humans is costly in supervision, computation, and interpretability; existing LLM prompting approaches lack systematic querying and cannot combine natural language with conventional pairwise trajectory feedback.

Approach

Combine LLM-derived prior over linear concept-weight preference functions (from language feedback) with Bradley-Terry likelihood of pairwise trajectory rankings in a Bayesian posterior, and select active queries via variance ratio filtered by an LLM oracle that predicts whether the human can answer.

Key results

• On the OSM routing environment, MAPLE with OAQS reached QSR 0.58 versus 0.43 for Active B-REX and 0.43 for MAPLE without OAQS.
• MAPLE outperformed B-REX on both Clear and Natural instruction datasets in the OSM routing domain, with B-REX approaching MAPLE accuracy as feedback increased.
• In HomeGrid, Mistral-Large-2 models beat B-REX by a wide margin, while GPT-4-mini performed worse than B-REX on cost difference.

Measured effects reported in the paper

Surprising findings

• Language feedback alone appeared nearly sufficient to align the preference direction, with cosine distance static as queries increased; preference feedback mainly calibrates magnitude.
• GPT-4-mini performed worse than B-REX in HomeGrid due to highly misaligned inferred weights, while the small local Mistral-7B was competitive with GPT-4o in routing.

Limitations

• Experiments use simulated human responses from templates and probabilistic rules rather than real users; user studies are future work.
• Effective query success rate for OAQS was lower than theoretical prediction because the variance-ratio acquisition selects harder queries, violating the independence assumption.
• Handling of unknown concepts is explicitly left beyond the scope of the paper.

Experimental setup

Datasets: OpenStreetMap road network data HomeGrid Minigrid environment trajectories (60 manually generated)

Benchmarks: OpenStreetMap routing benchmark HomeGrid

Metrics: cosine distance to ground-truth weights preference prediction accuracy policy cost difference query success rate (QSR) effective query success rate (EQSR)

Baselines: B-REX Active B-REX

Builds on: B-REX REVEALE Bradley-Terry model variation ratio active learning

Closest papers by meaning

• Deep Bayesian Active Learning for Preference Modeling in Large Language Models (NeurIPS 2024 · similarity 0.91)
• Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.89)
• TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.88)
• Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees (NeurIPS 2024 · similarity 0.87)
• RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.87)
• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.87)