Research map / Large language models / Language, safety and interpretability

Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Speech and audio language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that speech and text representations in speech LLMs align geometrically in middle layers (CKA) yet late-layer answer margins still diverge, with a fixed-input attention edit and compression experiments probing the gap on SpeechMMLU and VoiceBench BBH.

Problem

Speech LLMs answer semantically matched questions less accurately from speech than from text (the modality gap). It is unclear whether geometric alignment of speech and text representations implies that the model forms answers from them in the same way.

Approach

Combines cross-layer CKA between speech and text hidden states (with DTW token alignment) with layer-wise answer-margin readouts and a fixed-input attention intervention, showing that geometric alignment in middle layers does not imply functional equivalence in late-layer answer formation.

Key results

• In Qwen2.5-Omni on BBH, the late-layer crossover contrast in answer margins is +0.3652 with 95% interval [0.2867, 0.4426], despite middle-layer speech-text CKA alignment.
• Late speech attention sharpening increases absolute two-choice logit separation in all four mapping/rule-order cells (e.g., +.0291 identity/order 01), while text diffusion decreases it in all four.
• Tempo 1.2x repairs 61 of 178 speech-only errors and silence trimming repairs 68 in the overview protocol; token merging repairs 26.

Surprising findings

• Sharpening late speech attention increases answer-code separation without improving answer correctness, separating separation from accuracy.
• Geometric alignment is present in middle layers for Qwen2.5-Omni and MiniCPM-o while late-layer answer margins still differ by route.

Limitations

• Scope limited to semantics-matched reasoning over synthesized prompts and ~7B open-weight speech LLMs.
• Attention intervention tested only on Qwen2.5-Omni on BBH and does not establish accuracy repair or causal explanation of the full gap.
• Diagnostics do not identify which acoustic or phonetic features cause speech errors.

Experimental setup

Datasets: SpeechMMLU VoiceBench BBH MMSU

Benchmarks: SpeechMMLU VoiceBench BBH MMSU

Metrics: exact-match accuracy linear CKA candidate probability margin logit margin answer-code separation

Baselines: ASR-to-LLM cascade text-to-text (T2T) prompting

Builds on: logit lens tuned lens linear CKA

Closest papers by meaning

• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
• Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs (NeurIPS 2026 · similarity 0.88)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
• The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.86)
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.86)