Research map / Large language models / Alignment and preferences

Enhancing Speech Large Language Models through Reinforced Behavior Alignment

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VIRBA, an RL framework that aligns SpeechLMs across multiple TTS-generated acoustic views of the same instruction using verifiable, invariance, preference, and adaptive reasoning rewards optimized with CA-GRPO.

Problem

SpeechLMs lag text LLMs on instruction following and reasoning, and their behavior drifts across speakers, accents, noise, prosody, and disfluencies for the same semantic request.

Approach

Makes the multi-view speech group (K acoustic realizations of one instruction, N sampled responses each) the unit of RL optimization, with a group-relative advantage normalized over all views and samples, plus a cross-acoustic invariance reward and rule-verifiable rewards.

Key results

• VIRBA on Qwen2.5-Omni reaches 79.4 Spoken-Alpaca LC and 68.2 Speech-IFEval vs 71.8 and 60.7 for the Qwen2.5-Omni base, with 65.0 on disfluency LC.
• VIRBA (Qwen2-Audio) improves SpeechR MC from 45.8 (base) to 62.8 and single-view RL reaches only 52.7 on the same benchmark.
• Full VIRBA reward reaches 0.71 Spearman correlation with expert preference vs 0.56 for LLM judge only, with length bias reduced to 0.08.

Surprising findings

• Step-Audio-R1 remains stronger than VIRBA on several audio reasoning benchmarks (SpeechR, SoundMind, MMAR), per the paper's own tables.
• Single-view RL underperforms the base Qwen2-Audio on SpeechR MC (52.7 vs 45.8 is higher, but Group-DPO and single-view RL show limited gains on several columns relative to VIRBA).

Limitations

• Synthetic TTS views may underrepresent strong regional accents, assistive speech, and code-switching.
• Judge and reward models can encode bias; rule-based verifiers do not eliminate risk.
• Results are from single runs with paired bootstrap intervals; repeated training runs not reported.

Experimental setup

Datasets: Spoken-Alpaca Speech-IFEval DOWIS VocalBench-DF WebQuestions Llama-Questions TriviaQA SpeechR MMAR SoundMind FLEURS CoVoST2

Benchmarks: Spoken-Alpaca Speech-IFEval DOWIS VocalBench-DF WebQuestions TriviaQA SpeechR SoundMind MMAR FLEURS CoVoST2 MuST-C

Metrics: length-controlled win rate accuracy exact-match/F1 BLEU COMET cross-view consistency robustness drop

Baselines: TTS-SFT Group-DPO Single-view RL GLM-4-Voice Kimi-Audio Audio Flamingo 3 Step-Audio-R1 Whisper-large V2 SeamlessM4T-L V2

Builds on: GRPO DPO Qwen2-Audio Qwen2.5-Omni CosyVoice Qwen3-TTS

Closest papers by meaning

• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.87)
• Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.87)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.87)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.87)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.87)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.87)