Research map / Large language models / Alignment and preferences
Advancing Audio-Based Text Generation with Imbalance Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Imbalanced Preference Optimization (IPO), which uses human or GPT-4 win-loss judgments between two audio-to-text or TTS models, expands them with beam-search pseudo-labels, and optimizes both models interactively to improve speech translation, audio captioning, and zero-shot TTS.
Problem
Preference alignment for audio-to-text and text-to-audio generation is largely unexplored, and no open preference dataset exists; audio annotation is costly, so data-efficient alignment with limited human feedback is needed.
Approach
Model adversarial sampling in which annotators judge which of two models wins on each audio input, beam search hypotheses inherit the win/loss pseudo-label, and a KTO-style unpaired loss with a reference point optimizes both models on imbalanced positive/negative sets so they update interactively.
Key results
- On FLEURS X→En (15 directions), IPO raises SeamlessM4T average BLEU from 29.4 to 32.4 and Whisper from 23.5 to 27.0, with iterative optimization reaching 33.2 for SeamlessM4T.
- On Clotho, 400 annotations raise Pengi SPIDEr from 0.271 to 0.291 and Qwen-Audio from 0.288 to 0.306.
- On LibriTTS zero-shot TTS, 200 annotations reduce VC-330M WER from 17.9 to 8.0 and raise its SIM from 0.76 to 0.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IPO to Whisper-large-v2 | Whisper-large-v2 · FLEURS X→En test (15-language average) | BLEU: 23.5 -> 27.0 | Helped |
| Apply IPO to SeamlessM4T-large-v2 | SeamlessM4T-large-v2 · FLEURS X→En test (15-language average) | BLEU: 29.4 -> 32.4 | Helped |
| Repeat sampling-learning optimization for 3 rounds (ablation) | IPO-Whisper and IPO-SeamlessM4T · FLEURS X→En test (15-language average) | BLEU: 27.0 -> 27.5; 32.4 -> 33.2 | Helped |
| Use unified finetuning across languages instead of language-specific finetuning (ablation) | IPO-Whisper and IPO-SeamlessM4T · FLEURS X→En test (15-language average) | BLEU: 27.0 -> 26.0; 32.4 -> 31.0 | Hurt |
| Apply IPO with 100 human annotations | Whisper-large-v2 and SeamlessM4T-large-v2 · FLEURS De→En test | BLEU: 34.6 -> 35.7; 37.1 -> 38.2 | Helped |
| Remove beam-search pseudo-preferences (ablation) | IPO with 400 human annotations · FLEURS De→En test | BLEU: 36.9 -> 35.9; 40.6 -> 38.8 | Hurt |
| Apply IPO with 400 human annotations | Pengi · Clotho | CIDEr: 0.416 -> 0.448 | Helped |
| Remove beam-search pseudo-preferences (ablation) | IPO-Pengi · Clotho | CIDEr: 0.448 -> 0.440 | Hurt |
| Apply IPO with 400 human annotations | Pengi · Clotho | SPICE: 0.126 -> 0.133 | Helped |
| Apply IPO with 400 human annotations | Qwen-Audio · Clotho | SPIDEr: 0.288 -> 0.306 | Helped |
| Apply IPO with 200 human annotations | VC-330M · LibriTTS | MOS: 3.37 -> 3.85 | Helped |
| Apply IPO with 200 human annotations | VC-330M · LibriTTS | WER: 17.9 -> 8.0 | Helped |
Surprising findings
- The stronger model (SeamlessM4T, Qwen-Audio, VC-830M) also improved substantially from IPO, not only the weaker one.
- Only about 100 human annotations yielded clear BLEU gains on AST, showing high data efficiency.
Limitations
- Human annotation of audio is labor-intensive, and the human-labeled data is small (a few hundred annotations).
- Beam-search pseudo-labels may introduce errors; the empirical validation is limited to BLEU variance on one model.
- GPT-4 as annotator relies on ground-truth source text and only applies to public datasets with transcripts.
Experimental setup
Datasets: FLEURS Clotho LibriTTS
Benchmarks: FLEURS Clotho LibriTTS
Metrics: BLEU CIDEr SPICE SPIDEr MOS WER SIM human A/B win rate
Baselines: Whisper-large-v2 SeamlessM4T-large-v2 Pengi Qwen-Audio VoiceCraft-330M VoiceCraft-830M
Builds on: KTO DPO PPO Whisper SeamlessM4T
Closest papers by meaning
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.87)
- SpeechAlign: Aligning Speech Generation to Human Preferences (NeurIPS 2024 · similarity 0.87)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.86)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.86)
- RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.