Research map / Vision-language and multimodal / Multimodal LLMs
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, Shaozuo Yu, Sitong Wu and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add latent cross-modality regularizer (LCMR) loss (ablation) | Lyra · MM-Vet (speech+image) | accuracy: 53.1 -> 58.1 | Improved |
| add LCMR loss (ablation) | Lyra · TextVQA (speech+image) | accuracy: 76.7 -> 77.8 | Improved |
| add LCMR loss (ablation) | Lyra · LibriSpeech | word error rate: 1.9 -> 2.0 | No clear effect |
| add LCMR loss (ablation) | Lyra · MM-Vet (text+image) | accuracy: 61.1 -> 62.6 | No clear effect |
| add latent multi-modality extractor LMME(4, 0.7) to long-context inference (ablation) | Lyra baseline · prefill at 2^15 tokens | prefill time: 2.99 -> 1.23 | Improved |
| add LMME so long inputs fit in memory (ablation) | Lyra baseline · 2^16 tokens | memory: OOM -> 33G | Improved |
| add LMME(4, 0.7) to training on Lyra-LongSpeech-12K (ablation) | Lyra · training time | hours: 9.6h -> 4.5h | Improved |
| add LMME(4, 0.7) to training on Lyra-MM-1.5M (ablation) | Lyra · training time | hours: 66h -> 41h | Improved |
| add extractor when training Vicuna-7B model from scratch (ablation) | Vicuna-7B CLIP+Conv baseline · average of six benchmarks | average rate: 100.0% -> 101.5% | Improved |
| add extractor when training Qwen2-7B on LLaVA-665K (ablation) | Qwen2-7B SigLIP baseline · average of six benchmarks | average rate: 100.0% -> 99.6% | No clear effect |
| add long speech capability integration to baseline (ablation) | Baseline 7B · VideoMME overall | accuracy: 62.8 -> 66.2 | Improved |
| replace SFT with multi-modality LoRA when adding speech (ablation) | Qwen2-VL · TextVQA | accuracy: 81.3 -> 82.6 | Improved |
Closest papers by meaning
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.88)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.88)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.88)
- OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.88)
- OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.87)
- ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decoding (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
