Research map / Large language models / Retrieval and knowledge
Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Spectron adapts a pre-trained PaLM-2-style LM (350M/1B) with a USM Conformer speech encoder to do spoken QA and speech continuation directly on spectrograms, trained end-to-end with a joint CE (ASR + text continuation) and spectrogram derivative regression loss.
Problem
Spoken language models typically rely on discrete speech tokens and cascaded or multi-stage systems, limiting speaker preservation, semantic coherence and transfer of knowledge from text LLMs to speech input/output.
Approach
Connect a pre-trained speech encoder to a pre-trained prefix LM via a projection layer, and train end-to-end on continuous spectrograms with pre/post-net MLPs, a joint cross-entropy loss over transcript and text continuation, and a spectrogram regression loss including feature and time derivatives, so the model produces an intermediate text "chain-of-thought" before speech.
Key results
- Spectron (350M) reaches log-perplexity 126.08 vs. 136.42 for SpeechGPT and 296.99 for GSLM on LibriSpeech 3-second-prompt continuations (lower is better).
- Spectron (1B) attains 22.9% accuracy on LLaMA-Questions and 6.1% on WebQuestions, versus SpeechGPT's 21.9% and 6.5%.
- Spectron (350M) achieves average speaker similarity 0.42, above AudioLM 3-RVQ (0.37) and SpeechGPT (0.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spectron instead of SpeechGPT | spoken language models · LibriSpeech test-clean | log-perplexity: 136.42 -> 126.08 | Helped |
| Use Spectron instead of AudioLM (12-RVQ) | spoken language models · LibriSpeech test-clean | N-MOS: 3.87 $\pm$ 0.32 -> 3.68 $\pm$ 0.29 | Hurt |
| Use Spectron instead of AudioLM (3-RVQ) | spoken language models · LibriSpeech test-clean | speaker similarity: 0.37 -> 0.42 | Helped |
| Use Spectron instead of SpeechGPT (7B) | spoken QA models · LLaMA-Questions | accuracy: 21.9 -> 22.9 | Helped |
| Use Spectron instead of SpeechGPT (7B) | spoken QA models · Web Questions | accuracy: 6.5 -> 6.1 | Hurt |
| Add text cross-entropy loss (ablation) | Spectron · LibriSpeech test-clean | log-perplexity: 714.43 -> 126.08 | Helped |
| Add feature- and time-delta spectrogram losses (ablation) | Spectron · LibriSpeech test-clean | log-perplexity: 787.89 -> 126.08 | Helped |
| Use a pre-trained language model instead of training it from scratch (ablation) | Spectron · LibriSpeech test-clean | log-perplexity: 201.71 -> 126.08 | Helped |
| Use a pre-trained speech encoder instead of training it from scratch (ablation) | Spectron · LibriSpeech test-clean | log-perplexity: 213.25 -> 126.08 | Helped |
| Use pre-trained language model and speech encoder instead of training both from scratch (ablation) | Spectron · LibriSpeech test-clean | log-perplexity: 244.39 -> 126.08 | Helped |
Surprising findings
- Removing the cross-entropy text loss raised log-perplexity from 126.08 to 714.43, the largest ablation effect.
- Zero-shot Spectron matches SpeechGPT on spoken QA despite using a 1B LM versus a 7B LM with instruction tuning.
Limitations
- Generating spectrogram frames at 12.5 ms rate is expensive in time and memory, so long utterances are not feasible.
- Text and spectrogram decoding are sequential, hindering streaming and adding latency.
- Biases from the pre-trained language model may carry over.
Experimental setup
Datasets: Libri-Light LibriSpeech test-clean WebQuestions
Benchmarks: LibriSpeech test-clean WebQuestions LLaMA-Questions
Metrics: log-perplexity N-MOS speaker similarity answer accuracy
Baselines: GSLM AudioLM TWIST SpeechGPT
Builds on: PaLM 2 Conformer USM WaveFit SpecAugment
Closest papers by meaning
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.87)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.86)
- Can Speech LLMs Think while Listening? (ICLR 2026 · similarity 0.86)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.85)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.84)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.