Research map / Vision-language and multimodal / Multimodal LLMs

Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a plug-and-play unified completeness network (feature approximation via cross-/intra-modal neighbor graphs, teacher-student distillation, and noun-phrase-weighted contrastive integration) that makes video-language models robust to missing frames and words across retrieval, VideoQA, and video sentence grounding.

Problem

Video-language models assume complete video and text inputs, but real-world data often has missing frames and words at unbalanced rates, causing large performance drops in video-text retrieval, VideoQA, and video sentence grounding.

Approach

Approximate missing frame/word features using K-reciprocal nearest neighbors (Jaccard distance over cross- and intra-modal graphs) refined by a graph affinity matrix, distill a complete-pair teacher into an incomplete-pair student, and weight contrastive alignment by noun-phrase overlap so it can be plugged into existing task-specific models.

Key results

• On MSR-VTT text-to-video retrieval with incomplete pairs, X-Pool+Ours reaches R@1 36.3 versus 27.5 for X-Pool alone.
• On NExT-QA with incomplete pairs, BLIP-2+Ours improves temporal accuracy from 36.8 to 49.0.
• On TACoS, Ours+2D-TAN achieves R@1 IoU=0.5 of 26.76 at 0.63s runtime, faster than MomentDiff (21.40, 1.85s).

Surprising findings

• Plug-in gains on incomplete inputs are very large (e.g., over 8 R@1 points on MSR-VTT), while complete-pair gains are modest.

Limitations

• Feature completion via a pre-trained LMM and neighbor graphs is described as time-consuming for long videos, motivating distillation.
• Evaluation uses a default 30% missing rate, and the reported incomplete-setting results depend on the chosen missing-pattern simulation.

Experimental setup

Datasets: MSR-VTT LSMDC ActivityNet Captions Charades-STA TACoS NExT-QA STAR

Benchmarks: MSR-VTT LSMDC ActivityNet Captions Charades-STA TACoS NExT-QA STAR

Metrics: R@1 R@5 R@10 R@n IoU VideoQA accuracy

Baselines: X-Pool DiffusionRet CLIP-ViP T-MASS UATVR All-in-One MIST HiTeA InternVideo BLIP-2 SeViLA MMN

Builds on: X-Pool DiffusionRet CLIP-ViP T-MASS 2D-TAN

Closest papers by meaning

• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.89)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.88)
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (CVPR 2024 · similarity 0.87)
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.87)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.87)