Research map / Vision-language and multimodal / Multimodal LLMs
MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MCIF, a human-annotated benchmark of English scientific talks (ACL 2023) with German, Italian and Chinese parallel translations, evaluating 23 LLMs, SpeechLLMs, VideoLLMs and MLLMs across speech, video and text on short and long inputs.
Problem
Existing instruction-following benchmarks rarely jointly cover crosslingual, multimodal (speech, video, text), long-form inputs, and human annotation, leaving no unified evaluation of MLLMs on these dimensions.
Approach
A fully parallel, human-curated benchmark built from scientific talk videos, with transcripts, translations, abstracts-as-summaries and QA pairs in four languages, paired with fixed and randomly sampled instruction prompts to measure robustness to prompt wording.
Key results
- Phi4-Multimodal reaches 6.8 WER on short-form recognition, among the best recognition scores, while several models exceed 100 WER.
- Gemini 2.5 Flash gives the best long-form question answering (BERTScore 46.1) and long-form recognition is challenging for most systems.
- Summarization is the hardest macro-task, with some models producing negative BERTScores (e.g., MiniCPM-o-2 and LLaVA-NeXT-Video worse than random).
Surprising findings
- Ola drops from 6.6 WER on long-form recognition to 98.8 on short-form, often captioning slides instead of transcribing.
- Joint speech+video input often fails to outperform speech alone, and video-only QA underperforms speech despite video performing well in some short-form cases.
Limitations
- Small test set: 21 talks for most tasks plus 79 additional videos for summarization, about 10 hours of content.
- Only four languages (English, German, Italian, Chinese) and scientific-talk domain.
- Restricted to open-weight models under 20B parameters plus one commercial model, due to compute constraints.
Experimental setup
Datasets: ACL 2023 presentation videos
Benchmarks: MCIF
Metrics: WER COMET BERTScore (rescaled)
Baselines: DeSTA2 GraniteSpeech 3.3 Phi4-Multimodal Qwen2-Audio UltraVox v0.5 InternVL3 LLaVA-NeXT-Video Qwen2.5-VL VideoLLaMA3 Video-XL-2 Gemma 3n Ming-Lite-Omni
Builds on: Whisper normalizer COMET BERTScore SHAS
Closest papers by meaning
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.91)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.90)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.90)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.