Research map / Video / Video-language understanding
OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
JUNFU PU, Yuxin Chen, Teng Wang, Ying Shan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought plot and character reasoning before decoding (ablation) | OmniScript SFT · V2S event-level | Overall F1: 35.3 -> 37.0 | Improved |
| add RL after SFT with CoT (ablation) | OmniScript · V2S event-level | Overall F1: 37.0 -> 37.7 | Improved |
| use temporally segmented reward instead of global reward (ablation) | OmniScript GRPO · V2S event-level | Overall F1: 37.0 -> 37.7 | Improved |
| drop CoT when using segmented-reward RL (ablation) | OmniScript SFT+RL · V2S event-level | Overall F1: 37.0 vs 37.7 with CoT | No clear effect |
| add audio input and audio-injection pretraining (ablation) | Qwen3-VL-8B-SFT · V2S event-level | Dialogue F1: 52.0 -> 68.2 | Improved |
| add audio input and audio-injection pretraining (ablation) | Qwen3-VL-8B-SFT · V2S event-level | Overall F1: 31.1 -> 35.3 | Improved |
| add audio input (ablation) | Qwen3-VL-8B-SFT · V2S event-level | tIoU@0.1: 68.2 -> 66.6 | Hurt |
| mask subtitles at inference (ablation) | OmniScript-8B · V2S event-level | Dialogue F1: 72.2 -> 63.8 | Hurt |
| mask subtitles at inference (ablation) | Qwen3VL-235B-A22B · V2S event-level | Dialogue F1: 58.6 -> 7.7 | Hurt |
| mask subtitles at inference (ablation) | Gemini-3-pro · V2S event-level | Dialogue F1: 68.8 -> 60.9 | Hurt |
| use thinking variants of base models | Qwen3VL · V2S event-level | Overall F1: Qwen3VL-32B 32.3 -> 22.0 (thinking) | Hurt |
| use two-stage script generation instead of long-context extension on 30-45 min videos | OmniScript · V2S long videos | F1 | Improved |
Closest papers by meaning
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.87)
- MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.86)
- JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation (NeurIPS 2025 · similarity 0.86)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.86)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
