Research map / Vision-language and multimodal / Multimodal LLMs
LLark: A Multimodal Instruction-Following Language Model for Music
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLark, an instruction-tuned multimodal music LLM built from a generative Jukebox encoder, a Llama 2 chat model, and a linear projector, trained on about 1.2M instruction pairs generated from augmented open music datasets.
Problem
Existing audio-language models lack broad, fine-grained music understanding, largely because high-quality richly annotated music instruction data is scarce and music-specific multitask models are limited.
Approach
Augment open music datasets with estimated tempo, key, chords and beats, convert annotations into unified instruction-response pairs via GPT-3.5/GPT-4, and combine a generative Jukebox encoder (mean-pooled to 10Hz) with Llama 2-7B-chat through a single linear projection trained for instruction following.
Key results
- On GiantSteps key estimation, LLark scores 0.70 MIREX versus 0.32 for the Essentia baseline and 0.743 for the task-specific fine-tuned SOTA.
- LLark reaches 0.86 Acc2 on GiantSteps tempo, above the other music-text models but below the fine-tuned SOTA at 0.925.
- In human win-rate captioning comparisons, LLark wins at 100% against LTU-AS, WAC and LP-MusicCaps on MusicNet and FMA, and at 99.6% or higher on MusicCaps against LP-MusicCaps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Jukebox-5B audio encoder with CLAP (ablation) | LLark · Music understanding benchmarks | Task performance | Hurt |
| Replace Llama 2-7B language model with MPT-1B (ablation) | LLark · Music understanding benchmarks | Task performance | Hurt |
| Replace Llama 2-7B language model with MPT-1B (ablation) | LLark · Tempo estimation | Acc2 | Hurt |
| Increase training pairs from 1% toward 50% of the pool (ablation) | LLark · Music understanding, captioning, and reasoning evaluations | Task performance | Helped |
| Use LLark instead of existing multimodal music-text models | GiantSteps key estimation · GiantSteps | MIREX Score: 0.048 -> 0.70; 0.00 -> 0.70 | Helped |
| Use LLark instead of existing multimodal music-text models | GiantSteps tempo estimation · GiantSteps | Acc2: 0.05 -> 0.86; 0.00 -> 0.86 | Helped |
| Use LLark instead of ImageBind-LLM | GTZAN genre classification · GTZAN | Acc@1: 0.71 -> 0.56 | Hurt |
| Use LLark instead of existing multimodal baselines | MusicNet instrument ID · MusicNet | F1: 0.86 -> 0.99 | Helped |
| Use LLark instead of existing multimodal baselines | MedleyDB genre classification · MedleyDB | Acc@1: 0.57 -> 0.56; 0.378 -> 0.56 | Mixed |
| Use LLark instead of captioning baselines | MusicNet and FMA captioning · MusicNet and FMA | Human caption preference | Helped |
| Use LLark instead of captioning baselines | MusicCaps captioning · MusicCaps | GPT-4 musical-detail win rate: 1.000 vs LAv2; 1.000 vs WAC; 1.000 vs LTU; 0.9962 vs LP-MC | Helped |
| Use LLark instead of IB-LLM and LTU-AS | Reasoning tasks · MusicNet, FMA, MTG-Jamendo, and MagnaTagATune | GPT-4 musical-detail win rate: 0.572414; 0.721689; 0.681382; 0.695402 vs IB-LLM; 0.905405; 0.888031; 0.907336; 0.901354 vs LTU-AS | Helped |
Surprising findings
- Genre classification on GTZAN is won by ImageBind-LLM (0.71) over LLark (0.56), which the authors attribute to possible exposure of ImageBind to popular music and genre tags.
- Ablating the Jukebox encoder for CLAP causes large music-understanding drops, and swapping Llama 2 for MPT-1B mainly hurts tempo estimation, suggesting numeric tokenization matters.
Limitations
- Limited to 25-second audio context from the Jukebox encoder.
- Human evaluators are non-expert and GPT-4 judging may carry biases.
- Training data limited to open-source, permissively licensed music, so weights and instruction data cannot be released due to audio licenses.
Experimental setup
Datasets: MusicCaps YouTube8M-MusicTextClips MusicNet FMA MTG-Jamendo MagnaTagATune GiantSteps GTZAN MedleyDB
Benchmarks: GiantSteps GTZAN MedleyDB MusicNet MusicCaps FMA
Metrics: MIREX score Acc2 Acc@1 F1 human win rate GPT-4 win rate audio-text matching accuracy
Baselines: ImageBind-LLM LTU-AS Whisper Audio Captioning LP-MusicCaps CLAP-based ablation MPT-1B ablation
Builds on: LLaVA Jukebox Llama 2 ImageBind-LLM LTU-AS
Closest papers by meaning
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction (ICML 2026 · similarity 0.88)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.87)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.87)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.87)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.87)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.