Research map / Vision-language and multimodal / Multimodal LLMs
JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JavisGPT, a unified MLLM for joint audio-video comprehension and generation that pairs a Qwen2.5-VL-7B backbone and SyncFusion audio-video module with a frozen JavisDiT generator via learnable queries, trained in three stages on a new 200K-sample instruction dataset.
Problem
Existing multimodal LLMs treat audio and video as separate channels and cannot jointly comprehend and generate temporally synchronized sounding videos, and pipeline-based audio-video generation suffers from desynchronization.
Approach
A SyncFusion module uses cross-attention to inject temporally aligned audio tokens into per-frame visual patch tokens, and synchrony-aware learnable queries (semantic and spatiotemporal-prior) map LLM hidden states into the condition space of a frozen JavisDiT generator, with alignment and diffusion losses.
Key results
- On audio-video QA, JavisGPT reports 93.8 on AVQA and 82.1 on MU-AVQA, above Qwen2.5-Omni (91.5, 79.9), using 1.5M training samples.
- On JavisBench-mini generation, JavisGPT reaches FVD 317.5 and JavisScore 0.157, slightly above the JavisDiT base generator (327.8 and 0.153).
- SyncFusion matches or exceeds concatenation and interleaving on AVQA, MU-AVQA and AVSD with 2.0K tokens versus 3.5K and lower latency (224ms vs 246ms concat, 555ms interleave).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SyncFusion instead of concatenation | JavisGPT AV comprehension · AVQA | accuracy: 93.3 -> 93.4 | No clear effect |
| Use SyncFusion instead of interleaving | JavisGPT AV comprehension · AVQA | accuracy: 93.3 -> 93.4 | No clear effect |
| Use SyncFusion instead of Q-Former | JavisGPT AV comprehension · AVQA | accuracy: 71.4 -> 93.4 | Helped |
| Add audio-video fine-tuning (ablation) | JavisGPT generation · Audio-video generation | JavisScore: 0.069 -> 0.157 | Helped |
| Add multimodal pretraining (ablation) | JavisGPT generation · Audio-video generation | JavisScore: 0.135 -> 0.157 | Helped |
| Add audio-video fine-tuning (ablation) | JavisGPT generation · Audio-video generation | FVD: 537.4 -> 317.5 | Helped |
| Add multimodal pretraining (ablation) | JavisGPT generation · Audio-video generation | FVD: 349.6 -> 317.5 | Helped |
| Add audio-video fine-tuning (ablation) | JavisGPT generation · Audio-video generation | FAD: 9.6 -> 7.6 | Helped |
| Add multimodal pretraining (ablation) | JavisGPT generation · Audio-video generation | FAD: 8.4 -> 7.6 | Helped |
Surprising findings
- Aligning LLM outputs to DiT's original text encoder alone (MM-PreTrain only) gives poor generation (JavisScore 0.069), showing the alignment stage alone is insufficient.
- Joint training of comprehension and generation improves both sides, with the larger gain on generation.
Limitations
- Built on a 7B backbone and public datasets; scalability to larger LLMs not explored.
- Misaligned training objectives (NTP vs diffusion) and asymmetric input-output modeling for the shared LLM.
- Audio-video synchrony improves only marginally from joint training (+0.04).
Experimental setup
Datasets: TAVGBench AudioSet AudioCaps VGGSound WavCaps Clotho ESC50 GTZAN MACS UrbanSound8K VideoLLaMA2 AV-QA data LLaVA-Video-178K
Benchmarks: ActivityNet-QA Perception Test MVBench ClothoAQA TUT2017 AVQA MU-AVQA AVSD JavisBench-mini
Metrics: accuracy FVD KVD FAD TV-IB TA-IB CLIP score CLAP score AV-IB CAVP
Baselines: NExT-GPT UnifiedIO-2 Qwen2.5-Omni VideoLLaMA2.1 Macaw-LLM AV-LLM VideoLLaMA MM-Diff JavisDiT Qwen2-Audio Qwen-Audio Video-LLaVA
Builds on: Qwen2.5-VL JavisDiT BEATs MetaQueries VideoLLaMA2 LoRA
Closest papers by meaning
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.88)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.88)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.