Research map / Video / Video generation
SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SafeSora, a human preference dataset for text-to-video generation decoupling helpfulness and harmlessness, and demonstrates its use for moderation, reward/cost modeling, and Best-of-N fine-tuning of a prompt refiner and diffusion model.
Problem
Text-to-video generation lacks large human preference datasets for aligning outputs with human values, especially on harmlessness conditioned on text-video combinations.
Approach
Collects T-V pair preferences with decoupled helpfulness (4 sub-dimensions) and harmlessness (12 harm categories) annotations via a two-stage heuristic process, and validates the data by training a moderation model, reward/cost models, and BoN-fine-tuned prompt refiner and diffusion model.
Key results
- T-V Moderation reaches 82.94% agreement with human judgment on the test set.
- Reward model and cost model reach 65.29% and 72.41% agreement with crowdworkers respectively.
- GPT-4o shows low agreement with crowdworkers on preference labels, approaching random (0.5) as frames increase, but high agreement on harm labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train T-V Moderation on SafeSora multi-label data | Video-LLaVA · SafeSora test set | agreement with human judgment: 82.94% | Helped |
| Train a reward model on helpfulness preferences | Preference predictor · SafeSora preference evaluation set | agreement with crowdworkers: 65.29% | Helped |
| Train a cost model on harmlessness preferences | Preference predictor · SafeSora preference evaluation set | agreement with crowdworkers: 72.41% | Helped |
| Use GPT-4o frame comparisons for preference labeling instead of crowdworker judgments | GPT-4o preference evaluation · SafeSora evaluation set | agreement with crowdworkers: 0.5 | Hurt |
| Prioritize instruction following among helpfulness criteria | Crowdworker helpfulness judgments · SafeSora | correlation with overall helpfulness preference: 0.85 | Helped |
| Align on helpfulness for potentially harmful prompts | Text-to-video generation alignment · Potentially harmful prompts in SafeSora | helpfulness preferences contradicting harmlessness preferences: 53.39% | Mixed |
Surprising findings
- GPT-4o agreement with crowdworkers on overall and sub-dimension preferences is low and approaches random as more frames are used.
- 53.39% of helpfulness preferences on potentially harmful prompts contradict harmlessness preferences.
Limitations
- Baseline alignment algorithms (BoN fine-tuning) are not efficient enough as alignment methods.
- Prompt coverage cannot span all usage scenarios and misuse patterns.
- Human preference data reflects individual and group differences; GPT-4o cannot yet replace human annotators.
Experimental setup
Datasets: VidProM LAION-400M midjourney-detailed-prompts
Benchmarks: SafeSora-Eval
Metrics: agreement ratio with crowdworkers win rate harm label agreement
Baselines: LlamaGuard QA-Moderation GPT-4o
Builds on: Bradley-Terry model Best-of-N fine-tuning Video-LLaVA 3H standards (helpful, harmless, honest)
Closest papers by meaning
- VideoDPO: Omni-Preference Alignment for Video Diffusion Generation (CVPR 2025 · similarity 0.90)
- SafeVid: Toward Safety Aligned Video Large Multimodal Models (NeurIPS 2025 · similarity 0.89)
- $\mathcal{V}ista\mathcal{DPO}$: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models (ICML 2025 · similarity 0.88)
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.88)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.88)
- T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.