Research map / Large language models / Alignment and preferences

Same Words, Different Judgments: How Preferences Vary Across Modalities

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled cross-modal study comparing human and GPT-4o synthetic preference ratings for text vs. TTS-audio responses (100 PRISM prompts, ~94 US raters), finding comparable ICC reliability but modality-dependent decision thresholds, length bias, and near-chance cross-modal agreement.

Problem

Preference-based RL evaluation protocols were designed for text and have not been validated for speech; it is unknown whether audio preference ratings are as reliable as text and whether text-derived TTS preference data transfers to audio.

Approach

Matched text and TTS-audio evaluations of identical PRISM conversations with continuous VAS ratings and counterbalanced order, using ICC to quantify the number of raters needed and synthetic GPT-4o/GPT-4o-Audio ratings to predict human agreement.

Key results

• ICC(2,k) reached about .82 for audio and .79 for text with roughly 9 raters, while single-rater ICC(2,1) was poor (.333 audio, .295 text); the difference between modalities was not significant.
• Audio raters showed narrower decision thresholds (mean gap 27.9 vs 41.7 for text on decisive trials) and a weaker length effect (interaction p = .045).
• Audio and text agreed on the per-prompt winner for only 53% of prompts (binomial p = .31), and AI-human absolute error was lower for audio (12.27 vs 14.29).

Surprising findings

• Cross-modality winner agreement was at chance (53%), with modality effects varying by prompt rather than shifting ratings uniformly.
• Synthetic AI rating differentiation between two responses predicted higher human inter-rater agreement in both modalities.

Limitations

• Single static TTS voice (af_heart) and TTS-rendered speech lacks paralinguistic naturalness.
• Sample limited to US-based, mostly native English speakers; excludes safety and refusal scenarios.
• Synthetic ratings come from a single model family (GPT-4o and GPT-4o-Audio-Preview).

Experimental setup

Datasets: PRISM

Benchmarks: PRISM

Metrics: ICC(2,1) ICC(2,k) Krippendorff's alpha mean absolute error cross-modality agreement Mann-Whitney U

Baselines: GPT-4o GPT-4o-Audio-Preview PRISM original interactive ratings

Builds on: PRISM Kokoro Bradley-Terry RLAIF

Closest papers by meaning

• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.88)
• Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.88)
• Inverse Constitutional AI: Compressing Preferences into Principles (ICLR 2025 · similarity 0.88)
• Towards Understanding Sycophancy in Language Models (ICLR 2024 · similarity 0.88)
• Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.88)