Research map / Video / Video generation
VMDT: Decoding the Trustworthiness of Video Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VMDT, a unified trustworthiness benchmark for text-to-video and video-to-text foundation models across safety, hallucination, fairness, privacy, and adversarial robustness, evaluating 7 T2V and 19 V2T models.
Problem
Video foundation models lack a comprehensive, unified trustworthiness benchmark covering both T2V generation and V2T understanding, unlike text and image modalities.
Approach
Extends the MMDT image-modality trustworthiness framework to video, adding video-specific risks (temporal risk, physical harm), a temporal hallucination scenario, and video-based fairness, privacy (location inference, memorization), and adversarial datasets, evaluated with GPT-4o and Qwen2.5-VL-72B judges.
Key results
- Luma achieved the highest average T2V trustworthiness score of 70.1, while CogVideoX-5B scored lowest at 55.7.
- InternVL2.5-78B achieved the top V2T average trustworthiness score of 72.7, and Qwen2.5-VL-3B ranked lowest at 65.3.
- Within V2T models, privacy location inference correlates positively with model size (Pearson 0.544, P-value 0.016).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use larger models within the same V2T model class | V2T models · hallucination scenarios | accuracy | Helped |
| Use larger models within the same V2T model class | V2T models · adversarial inputs | robustness: P-value = 0.034 | Helped |
| Use larger models within the same V2T model class | V2T models · fairness | unfairness | Hurt |
| Use larger V2T models | V2T models · location inference | privacy inference score: Pearson Corr = 0.544, P-value = 0.016 | Hurt |
| Use larger V2T models | V2T models · overkill fairness | overkill fairness O | Helped |
| Evaluate T2V models with transformed rather than vanilla safety prompts | T2V models · safety | HGR | Helped |
| Use gradient-based rather than greedy or genetic adversarial attacks | T2V models · adversarial robustness | average performance drop: 5.0% vs -0.9% and 0% | Hurt |
| Use greedy rather than gradient-based adversarial attacks | T2V models · adversarial robustness | average performance drop: -0.9% | No clear effect |
| Use genetic rather than gradient-based adversarial attacks | T2V models · adversarial robustness | average performance drop: 0% | No clear effect |
| Generate safety prompts in transformed scenarios rather than vanilla scenarios | T2V models · safety | HGR | Helped |
| Scale up a V2T model | V2T models · trustworthiness dimensions overall | trustworthiness | Mixed |
Surprising findings
- Safety of V2T models shows no correlation with model size, while unfairness and privacy risks rise with scale.
- T2V models generate less harmful content in transformed scenarios, likely reflecting capability limitations rather than improved safety.
Limitations
- Closed-source model parameter sizes are unknown, so conservative lower-bound size assumptions are used in size correlation analyses.
- GPT and Claude process video via sampled frames rather than native video input.
- Safety and hallucination judgments rely on LLM judges with imperfect human agreement (about 86-88% for safety judges).
Experimental setup
Datasets: VATEX CLEVRER Neptune WebVid-10M BDD100k Casual Conversation First Impression V2 MMDT SafeWatch-Bench T2VSafetyBench
Benchmarks: VMDT
Metrics: bypass rate harmful content generation rate accuracy fairness score F1 fairness score F2 overkill fairness O l2 distance cosine similarity performance drop Pearson correlation
Baselines: VideoCrafter2 CogVideoX-5B OpenSora 1.2 Vchitect-2.0 Luma Pika Nova Reel GPT-4o Claude-3.5-Sonnet Nova Pro
Builds on: MMDT DecodingTrust SafeWatch-Bench T2VSafetyBench
Closest papers by meaning
- MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
- T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (NeurIPS 2024 · similarity 0.88)
- MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models (ICLR 2025 · similarity 0.88)
- Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs (NeurIPS 2025 · similarity 0.88)
- Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression (ICML 2024 · similarity 0.86)
- T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.