Research map / Video / Video-language understanding

Learning Object State Changes in Videos: An Open-World Perspective

CVPR 2024 · Video › Video-language understanding · approach: Event and anomaly reasoning in video · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces an open-world formulation for temporally localizing initial, transitioning, and end states of object state changes in videos, with a VLM-supervised model (VIDOSC) and the HowToChange benchmark.

Problem

Video object state change (OSC) localization is limited to closed vocabularies of objects seen in training; models fail to generalize to novel objects undergoing the same state transition.

Approach

Uses LLaMA2 to mine OSC examples from ASR transcripts and CLIP/VideoCLIP textual state descriptions to generate pseudo-labels, then trains a temporal transformer over frozen InternVideo features with object-agnostic shared state labels and object-centric features, so that the model generalizes to unseen objects.

Key results

• VIDOSC reaches 0.84 state Prec@1 on known and 0.57 vs 0.49 for MultiTaskChange on ChangeIt, and leads all baselines on all three datasets.
• On ChangeIt (open-world), VIDOSC's state Prec@1 drops only 8% from known to novel OSCs, versus 19% for MultiTaskChange.
• On HowToChange, VIDOSC achieves 60.7% known and 58.2% novel state Prec@1, compared with 43.1% and 38.8% for MultiTaskChange.

Measured effects reported in the paper

Surprising findings

• Zero-shot VLM baselines like CLIP and VideoCLIP are competitive with or outperform some supervised self-supervised baselines on the open-world split.

Limitations

• Performance gap between known and novel OSCs remains (e.g., 8% drop on ChangeIt open-world).
• Models handle only one OSC category at a time; concurrent OSC processes are not addressed.
• Spatial understanding of OSCs is not integrated.

Experimental setup

Datasets: HowTo100M ChangeIt HowToChange

Benchmarks: ChangeIt ChangeIt (open-world) HowToChange

Metrics: state precision@1 action precision@1 F1 (%) precision (%)

Baselines: CLIP VideoCLIP InternVideo LookForTheChange MultiTaskChange

Builds on: LookForTheChange MultiTaskChange CLIP VideoCLIP HowTo100M

Closest papers by meaning

• Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.87)
• PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.87)
• OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.87)
• Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.87)
• Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.86)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.86)