Research map / Vision-language and multimodal / Multimodal LLMs

CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CCFQA, a parallel speech-text factual QA benchmark over 8 languages, evaluates MLLMs on cross-lingual and cross-modal consistency, and proposes LLM-SQA, a GemmaX2-9B-based speech QA model that transfers English QA ability to other languages with 5-shot training.

Problem

MLLMs give inconsistent factual answers across languages and between text and speech inputs, and existing factuality benchmarks cover mainly English text or vision, lacking multilingual speech evaluation.

Approach

A fully parallel speech-text factual QA benchmark across 8 languages (human-recorded speech, translated questions), plus a English-pivot few-shot transfer strategy that trains a speech LLM on English SQA and adapts to other languages with 5 examples per language.

Key results

• On XSQA, LLM-SQA reaches 51.4 F1 / 39.7 LLM accuracy averaged over 8 languages, versus 45.7 F1 / 38.6 for GPT-4o-mini-Audio and 38.5 F1 / 29.5 for Qwen2.5-Omni-7B.
• On SQA, LLM-SQA averages 52.0 F1 and 40.3 LLM accuracy, close to GPT-4o-mini-Audio's 47.7 F1 and 40.4 accuracy.
• Text QA is much stronger than speech QA for most open models: Qwen2.5-Omni-7B averages 46.5 F1 on text QA but 44.0 F1 on speech QA, with LLM accuracy dropping from 38.2 to 33.2.

Surprising findings

• Phi-4-Multimodal scores 21.0 F1 but only 5.7 LLM accuracy on XSQA, showing high-F1 answers that are factually wrong, and its special-token design sharply hurts cross-lingual SQA.
• GPT-4o-mini is the only model that stays consistent across languages and modalities (cross-lingual consistency 96.6 / 95.5), while Phi-4-Multimodal's cross-lingual consistency in LLM accuracy is only 25.9.

Limitations

• Benchmark covers only text and speech modalities; extension to vision is future work.
• English-centric pivot strategy introduces language bias toward English.
• Speech recordings for some languages (French, Russian, Cantonese) show higher ASR error rates due to proper nouns.

Experimental setup

Datasets: MKQA MOOCCubeX FLEURS

Benchmarks: CCFQA

Metrics: F1 LLM-judged accuracy consistency ratio

Baselines: GPT-4o-mini-Audio Phi-4-Multimodal Qwen2-Audio Qwen2.5-Omni-3B Qwen2.5-Omni-7B

Builds on: LLM-SRT Whisper Q-Former

Closest papers by meaning

• Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.88)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.87)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.87)
• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.87)