Research map / Large language models / Language, safety and interpretability

MATS: An Audio Language Model under Text-only Supervision

ICML 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains an audio-language LLM (MATS) using only text supervision via CLAP's shared space, with a Gaussian-noise training scheme and a k-means memory-based Santa mechanism at inference to bridge the CLAP modality gap.

Problem

Large audio-language models need large audio-language pair corpora, which are costly to collect; text-only audio captioning methods are narrow (sound-only captioning). The paper targets multi-task audio understanding (classification, captioning, QA) trained without audio.

Approach

Train a mapper and LoRA-adapted LLM on CLAP language embeddings (with Gaussian noise) and at inference substitute CLAP audio embeddings processed by Santa, which mixes the audio embedding with a weighted sum of clustered, similar memory-bank language embeddings to reduce the audio-language modality gap.

Key results

• MATS-LLaMA reaches 0.735 CIDEr on AudioCaps versus 0.697 for NoAudioCaptioning, and Santa outperforms other modal-transfer methods on AudioCaps and AIR-Bench.
• On MMAU test-mini average, MATS-LLaMA (7B) scores 44.5 on test vs. 32.8 for SALMONN (13B), per the paper's second-best claim.
• Zero-shot audio classification: MATS achieves best results on several benchmarks, e.g. DCASE 0.43 and TUT 0.49 (MATS-GPT2) vs. baselines, and BJO 0.73 for MATS-LLaMA.

Measured effects reported in the paper

Surprising findings

• MATS-LLaMA trained only on text performs comparably to the audio-supervised MATS-Audio on several tasks, and outperforms it on zero-shot classification.
• MATS-GPT2 scores very low on MMAU (9.8 avg test) despite competitive close-ended results, showing small LLMs fail on open-ended reasoning.

Limitations

• CLAP struggles to capture fine-grained audio information (e.g., counts of occurrences), hurting AIR-Bench performance.
• Open-ended reasoning is weaker for the GPT2 variant, and MATS trails some audio-supervised models on ClothoAQA B-ACC and MMAU.
• Evaluation relies on GPT-4 judging for AIR-Bench and on text-matching for classification, and the training dataset AudioTIA-5M is not released.

Experimental setup

Datasets: AudioTIA-5M AudioSet FSD50K VGGSound DCASE ESC-50 UrbanSound8K TUT SESA VS CREMA-D RAVDESS

Benchmarks: ESC-50 FSD50K DCASE TUT VGG US8K GTZAN BJO AudioCaps Clotho-v2 MusicCaps ClothoAQA

Metrics: ACC mAP CIDEr SPICE SPIDEr ROUGE-L BLEU-4 B-ACC GPT4 score

Baselines: Pengi LTU GAMA Qwen-Audio Qwen-Audio-Chat SALMONN Audio Flamingo Qwen2-Audio-Instruction NoAudioCaptioning PromptAAC DRCap LP-MusicCaps

Builds on: CLAP CapDec DeCap NoAudioCaptioning Pengi LoRA

Closest papers by meaning

• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.89)
• UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · similarity 0.89)
• Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models (ICLR 2026 · similarity 0.89)
• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.89)
• Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.88)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.88)