Research map / Large language models / Alignment and preferences

Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Model Using Implicit Feedback from Pre-training Demonstrations

ICLR 2025 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DPA-OMF, which builds preference rankings among a motion model's own rollouts using optimal-transport occupancy-measure distance to pre-training expert demonstrations, then aligns a 1M-parameter MotionLM with contrastive preference learning, without human annotation.

Problem

Post-training preference alignment of auto-regressive multi-agent motion generation models is costly because preference rankings require human annotation; prior alignment-from-demonstrations methods treat all model samples as unpreferred.

Approach

Use an optimal-transport occupancy-measure distance between each rollout and the pre-training expert demonstration to rank the reference model's own samples, producing graded preference pairs among model generations with zero human cost, then fine-tune with a multi-agent contrastive preference loss.

Key results

• DPA-OMF raises the 1M MotionLM composite WOSAC realism from 0.721 to 0.739 and map compliance from 0.815 to 0.867, outperforming AFD (0.714) and SFT baselines.
• Ablation: full feature set gives composite realism 0.739 vs 0.724 with collision only and 0.705 with comfort only; adversarial AFD reaches 0.720 with minADE 1.539 vs 1.413 for ours.
• Preference-scaling analysis shows small preference data can degrade the model, and larger data reduces over-optimization at a given KL budget; the aligned model still trails SMART (0.761) and BehaviorGPT (0.747).

Measured effects reported in the paper

Surprising findings

• Adversarial AFD with expert demonstrations as preferred samples barely changes the likelihood of preferred and unpreferred model samples, while the expert likelihood dominates.
• Using only progress or comfort features in the preference distance degrades realism due to increased collision rate, while collision-only features still give improvement.

Limitations

• Preference distance relies on hand-designed features, and its relation to human preference is not strictly monotonic.
• Aligned 1M model still underperforms larger SOTA models such as SMART and BehaviorGPT.
• Optimal transport must be solved per agent, making end-to-end or RL-based use more costly.

Experimental setup

Datasets: Waymo Open Motion Dataset WOSAC

Benchmarks: WOSAC

Metrics: composite realism kinematic interactive map compliance minADE classification accuracy

Baselines: AFD (adversarial alignment from demonstrations) SFT on top-ranked samples (SFT-OA) SFT with ADE ranking BehaviorGPT Trajeglish SMART

Builds on: direct preference optimization MotionLM alignment from demonstrations optimal transport reward learning

Closest papers by meaning

• Predictive Preference Learning from Human Interventions (NeurIPS 2025 · similarity 0.89)
• Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.88)
• Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization (NeurIPS 2024 · similarity 0.88)
• AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.88)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents (ICLR 2026 · similarity 0.87)
• On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.87)