Research map / Video / Video-language understanding

PercepCap: Video Captioner with Structured Spatio-Temporal Perception

NeurIPS 2026 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PercepCap, a video captioner that first emits a structured spatio-temporal perception trace (object boxes over time and timestamped events) and then writes the caption, trained with perceive-then-describe SFT and GRPO with perception and caption rewards on caption-anchored data.

Problem

Video captioning by MLLMs is caption-only, so perceptual errors in object location, tracking, and event timing are hidden in the final text and cannot be supervised directly; caption-aligned perception training data is also scarce.

Approach

Make the perception evidence (object trajectories with boxes and timestamped events) explicit as a structured trace generated before the caption, supervise it via SFT, and optimize it with decomposed rewards (IoU, temporal IoU, event semantics, caption object/event F1) using GRPO. Training data is built by generating a caption first, extracting its objects/events, and grounding them in the video s

Key results

• On DREAM-1K, PercepCap reaches event F1 33.9 vs 29.1 for the Qwen3-VL 8B baseline, and the best AutoDQ F1 among open-source models.
• On VidCapBench-AE, PercepCap achieves overall Accuracy 19.0 vs 17.8 for Qwen3-VL 8B, and best on most categories.
• On ShortVidBench, PercepCap reaches 76.1 accuracy vs 72.8 for Qwen3-VL 8B, though MotionBench (54.9) trails Qwen3.5 9B (56.6).

Surprising findings

• Simply prompting the caption generator with a perception chain from Gemini 3 Flash (zero-shot, no training) already raises DREAM-1K F1 from 29.1 to 33.5.

Limitations

• Supervision relies on annotator-generated (Gemini) perception data rather than manually verified tracks and boundaries, so errors can propagate.
• The perceive-describe chain increases inference cost and output length.
• The fixed trace schema may be less flexible for dense interactions, long videos, or domain-specific objects.

Experimental setup

Datasets: caption-aligned perception data built from 57K videos YouTube-VOS ActivityNet DREAM-1K CaReBench ShortVidBench MotionBench VidCapBench-AE

Benchmarks: DREAM-1K CaReBench ShortVidBench MotionBench VidCapBench-AE YouTube-VOS ActivityNet

Metrics: AutoDQ F1/Precision/Recall Action and Object F1/Recall/Precision QA accuracy Accuracy/Precision/Coverage/Conciseness mean bbox IoU SODA_m F1

Baselines: Qwen3-VL 8B Qwen3.5 9B Qwen2.5-VL 7B Kimi-VL A3B GLM 4.6v 10B Gemini 3 Flash Gemini 3.1 Pro GPT-4o MiMo-VL InternVL 8B

Builds on: GRPO DeepSeek-R1 VideoP2R Qwen3-VL DREAM-1K

Closest papers by meaning

• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.88)
• Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.88)
• Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.87)
• PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.86)
• Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.86)
• Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)