Research map / Large language models / Language, safety and interpretability

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Multilingual LLM data and culture · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes C-Gate, a speech-to-LLM bridge that represents each audio frame as a top-16 convex combination of a frozen LLM's input embedding rows, and trains it jointly on ASR, emotion and reasoning tasks with a frozen Whisper encoder and Qwen2.5-7B-Instruct.

Problem

Speech-to-LLM interfaces either force near-discrete token alignment (losing paralinguistic information) or learn unconstrained continuous vectors that drift off the frozen LLM's input embedding manifold and destabilize autoregressive decoding.

Approach

Constrain every speech bridge vector to lie in convex(E), the convex hull of the LLM's input embedding table, by a top-K Q-Former-style query over full vocabulary with values equal to raw embedding rows, so the representation stays on the LLM's trained manifold while remaining continuous.

Key results

• C-Gate-2T reduces LibriSpeech autoregressive WER from 7.76% to 4.78% (38.4% relative) while reaching 97.1% RAVDESS emotion accuracy.
• C-Gate-3T reaches 3.98% AR-WER, a 48.7% relative reduction over C-Gate-ASR, with emotion accuracy dropping to 90.5%.
• Causal interventions: zeroing waveform raises WER from 2.8% to 97.5%; shuffling bridge trajectory in time raises WER to 244.5%.

Surprising findings

• Pooled per-utterance bridge vectors reach only 16.8% emotion accuracy, yet a bag of selected token ids reaches 77.7%, exceeding the pre-bridge Whisper pool (67.8%).
• Per-frame routing posteriors are near uniform (normalized entropy 0.96-0.99), so information is carried by the selected-support trajectory rather than routing weights or token identity.

Limitations

• Only one encoder-LLM pair and a single training seed were evaluated.
• Emotion benchmark is acted, closed-set RAVDESS; reasoning probes lack audio-grounding audits, so reasoning gains are treated as boundary measurements.
• Gap to state-of-the-art audio LLMs is attributed to much smaller training scale; no matched learned-token Q-Former baseline is reported.

Experimental setup

Datasets: LibriSpeech RAVDESS VoiceBench-BBH BBH heldout MMSU MMAU SpeechMMLU

Benchmarks: LibriSpeech test-clean RAVDESS VoiceBench-BBH BBH-HO MMSU MMAU SpeechMMLU

Metrics: WER accuracy effective rank entropy KL divergence

Baselines: C-Gate-ASR C-Gate-Emotion C-Gate-Reasoning Qwen-Audio-Chat Qwen2-Audio-Instruct Qwen2.5-Omni-7B Kimi-Audio-7B-Instruct Audio Flamingo 3 SALMONN

Builds on: Q-Former (BLIP-2) dynamic loss reweighting Whisper

Closest papers by meaning

• Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.88)
• Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap (NeurIPS 2026 · similarity 0.88)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.87)
• MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.87)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.86)
• Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR 2024 · similarity 0.86)