Research map / Video / Video-language understanding
SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlowFocus, which grounds the query-relevant video segment with a Vid-LLM, densely resamples that segment, and fuses high- and low-frequency visual tokens via multi-frequency mixing attention, trained in three stages; also introduces the FineAction-CGR benchmark.
Problem
Video LLMs must trade off per-frame token quality against the number of sampled frames, so they miss brief query-relevant details needed for fine-grained temporal grounding and reasoning.
Approach
Two-pass inference: the Vid-LLM first predicts the query-related temporal segment from low-frequency frames, then densely samples that segment at high frequency. A multi-frequency mixing attention (cross-attention from high-frequency tokens to low-frequency tokens) and a discretized temporal token encoder combine the two streams, and a three-stage training curriculum (alignment, temporal boundary
Key results
- On FineAction-CGR, SlowFocus reaches 66.68 mIoU for temporal grounding and 53.10% accuracy on temporal reasoning, versus 35.35 mIoU... (LLaMA-VID retrained on stage 3: 22.38 mIoU, 24.81% accuracy).
- Mixed-frequency sampling plus temporal encoder and MMA raises reasoning accuracy from 30.25% (low-frequency only) to 53.10%.
- On MovieChat-1K global mode the method scores 58.6% accuracy, and on EgoSchema it reaches 39.7%, competitive with specialized long-video methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SlowFocus instead of VTimeLLM | Vicuna-7B video LLM · FineAction-CGR | mIoU: 27.69 -> 66.68 | Helped |
| use SlowFocus instead of LLaMA-VID† | Vicuna-7B video LLM · FineAction-CGR | temporal reasoning accuracy: 24.81 -> 53.10 | Helped |
| use SlowFocus instead of LLaMA-VID | Vicuna-7B video LLM · MSVD-QA | accuracy: 69.7 -> 70.1 | Helped |
| use SlowFocus instead of LLaMA-VID | Vicuna-7B video LLM · MSRVTT-QA | accuracy: 57.7 -> 58.3 | Helped |
| use SlowFocus instead of LLaMA-VID | Vicuna-7B video LLM · ActivityNet-QA | accuracy: 47.4 -> 48.4 | Helped |
| add mixed-frequency sampling (ablation) | SlowFocus with low-frequency frames only · FineAction-CGR | temporal reasoning accuracy: 30.25 -> 39.12 | Helped |
| add a temporal encoder (ablation) | mixed-frequency sampling with NH = 10 · FineAction-CGR | mIoU: 34.81 -> 57.26 | Helped |
| increase high-frequency frames from NH = 10 to NH = 20 (ablation) | SlowFocus with temporal encoder · FineAction-CGR | mIoU: 57.26 -> 64.92 | Helped |
| increase high-frequency frames from NH = 20 to NH = 40 (ablation) | SlowFocus with temporal encoder · FineAction-CGR | mIoU: 64.92 -> 65.03 | No clear effect |
| add MMA (ablation) | SlowFocus with mixed-frequency sampling and temporal encoder · FineAction-CGR | temporal reasoning accuracy: 51.68 -> 53.10 | Helped |
| add stage 2 boundary enhancement (ablation) | stage 1 pre-trained model · FineAction-CGR | mIoU: 0.11 -> 51.67 | Helped |
| add stage 3 SlowFocus adaptation (ablation) | stages 1+2 model · FineAction-CGR | mIoU: 51.67 -> 66.68 | Helped |
Surprising findings
- Performance stays relatively stable when global sampling frequency is decreased while total token count is held constant, provided high-frequency tokens are allocated to the query segment.
- Increasing the temporal token space from 1K to 10K slightly hurts performance (mIoU 66.68 to 63.74).
Limitations
- Spatial detail ambiguity remains because the method focuses on temporal resolution and does not address high-resolution frames.
- Not specifically trained on long-video data, so long-video performance is only competitive, not state of the art.
Experimental setup
Datasets: LCS-558K WebVid 2.5M InternVid-10M-FLT ActivityNet Captions FineAction MSVD-QA MSRVTT-QA ActivityNet-QA MovieChat-1K EgoSchema
Benchmarks: FineAction-CGR MSVD-QA MSRVTT-QA ActivityNet-QA MovieChat-1K EgoSchema
Metrics: mIoU R@0.3 R@0.5 R@0.7 BLEU-4 METEOR ROUGE CIDEr accuracy score
Baselines: LLaMA-VID VTimeLLM VideoChat Video-ChatGPT VideoLLaMA MovieChat LLaMA-Adapter FrozenBiLM
Builds on: LLaMA-VID VTimeLLM LLaVA Panda70M
Closest papers by meaning
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.92)
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · similarity 0.91)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.91)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.91)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.91)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.