Research map / Vision-language and multimodal / Visual question answering

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

AAAI 2025 · Vision-language and multimodal › Visual question answering · approach: Multimodal question answering datasets · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a modality importance score (MIS), estimated with GPT-4 Turbo over subtitle-only, video-only and combined inputs, to show VidQA benchmarks (TVQA, LifeQA, AVQA) are dominated by unimodal or modality-agnostic questions with few complementary ones.

Problem

Existing video question answering benchmarks often rely on a single modality, so models may not truly integrate video and text/audio; there is no quantitative measure of which modality is needed per question.

Approach

Define a modality importance score from accuracy differences across modality subsets, estimate it with an MLLM (GPT-4 Turbo) instead of human annotation, and use it to categorize questions as unimodal-biased, modality-agnostic, or complementary, then test VidQA models with feature permutation.

Key results

• Across TVQA, LifeQA and AVQA, 89.8% to 94.8% of questions are answerable from a single modality or are modality-agnostic, with only 0.6% to 2% complementary.
• Human study (4 participants, 197 TVQA questions) gave Fleiss' kappa 0.76 among annotators and Cohen's kappa of about 0.3 between human and MLLM categorization.
• Feature permutation shows an average accuracy drop of 51.7 points for subtitle permutation on subtitle-biased questions versus 1.0 point for video permutation on modality-agnostic questions; models mostly rely on subtitles.

Measured effects reported in the paper

Surprising findings

• No complementary questions were identified in the human-evaluated subset of TVQA questions.
• Merlot Reserve achieved only 18% accuracy on 300 generated complementary questions.

Limitations

• Relies on a single MLLM (GPT-4 Turbo) for MIS estimation, with limited visual processing and frame-limit constraints.
• Sampled subsets (about 6-10% of TVQA and AVQA) prevent definitive dataset-wide conclusions.
• Human study is small (four participants) with moderate agreement (kappa about 0.3) and background knowledge confounds.

Experimental setup

Datasets: TVQA LifeQA AVQA VGGSound

Benchmarks: TVQA LifeQA AVQA

Metrics: accuracy modality importance score percentage of question types

Baselines: Merlot Reserve FrozenBiLM Llama-VQA MiniGPT4-Video

Builds on: TVQA Merlot Reserve FrozenBiLM Flipped-VQA MiniGPT4-Video

Closest papers by meaning

• Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.89)
• MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.89)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.89)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)