Research map / Large language models / Language, safety and interpretability
CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CountLLM, the first LLM-based repetitive action counting framework that feeds video tokens from a periodicity transformer plus periodicity text prompts into Vicuna-7B, trained with progressive multimodal alignment and LoRA instruction tuning.
Problem
Repetitive action counting in videos relies on regression networks with limited representational capacity and supervised training on small fitness-focused datasets, causing overfitting and poor generalization to diverse or out-of-domain actions.
Approach
Reformulate repetitive action counting as video question answering: a periodicity transformer compresses video features into periodic tokens fed to a pre-trained LLM along with a periodicity description prompt, and the LLM outputs the count as a fixed-length decimal string, trained via progressive semantic, periodicity-text and instruction alignment.
Key results
- On RepCount with VideoMAE encoder, CountLLM reaches OBO 0.639 and MAE 0.162, versus ESCounts at 0.563 and 0.213.
- On Countix, CountLLM with VideoMAE achieves OBO 0.801 and MAE 0.215, versus ESCounts at 0.673 and 0.276.
- In cross-dataset RepCount-to-UCFRep, CountLLM reaches OBO 0.816 versus 0.571 for ESCounts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CountLLM with VideoMAE instead of ESCounts | RepCount test · RepCount test | MAE: 0.213 -> 0.162 | Helped |
| Use CountLLM with VideoMAE instead of ESCounts | UCFRep val · UCFRep val | OBO: 0.704 -> 0.839 | Helped |
| Use CountLLM with VideoMAE instead of ESCounts | Countix test · Countix test | OBO: 0.673 -> 0.801 | Helped |
| Add periodicity description to the instruction (ablation) | CountLLM · RepCount val | OBO: 0.476 -> 0.631 | Helped |
| Use decimal-string answers instead of learnable answer tokens (ablation) | CountLLM · RepCount val | OBO: 0.387 -> 0.631 | Helped |
| Add semantic-aware alignment (ablation) | CountLLM training · RepCount val | OBO: 0.368 -> 0.417 | Helped |
| Add periodicity-aware alignment (ablation) | CountLLM training · RepCount val | OBO: 0.368 -> 0.551 | Helped |
| Add both alignment stages progressively (ablation) | CountLLM training · RepCount val | OBO: 0.551 -> 0.631 | Helped |
| Add a periodicity transformer instead of using only a linear projector (ablation) | CountLLM · RepCount val | OBO: 0.260 -> 0.357 | Helped |
| Apply LoRA to the video encoder (ablation) | CountLLM · RepCount val | OBO: 0.357 -> 0.546 | Helped |
| Apply LoRA to the LLM (ablation) | CountLLM · RepCount val | OBO: 0.357 -> 0.498 | Helped |
| Apply LoRA to both the video encoder and LLM (ablation) | CountLLM · RepCount val | OBO: 0.546 -> 0.631 | Helped |
Surprising findings
- Video MLLMs such as VideoChat2, GPT-4V and VideoChatGPT fail to count out-of-domain periodic motions like clock hands and planetary orbits, while CountLLM does.
Limitations
- Substantial GPU resource demands due to the large LLM.
- Accuracy may improve with more precise periodicity descriptions, which were manually refined from GPT-4 output.
- Relies on fixed-length clip splitting for long videos.
Experimental setup
Datasets: WebVid-10M RepCount UCFRep Countix
Benchmarks: RepCount UCFRep Countix OVR-Kinetics
Metrics: OBO MAE
Baselines: RepNet Context Sight & Sound TransRAC Full MFL ESCounts OVRCounter VideoChat2 GPT-4V VideoChatGPT
Builds on: VideoChat2 Video Swin Transformer VideoMAE LoRA
Closest papers by meaning
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.85)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.85)
- Count What You Want: Exemplar Identification and Few-Shot Counting of Human Actions in the Wild (AAAI 2024 · similarity 0.84)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.83)
- UNICBench: UNIfied Counting Benchmark for MLLM (CVPR 2026 · similarity 0.83)
- LLMs are Good Action Recognizers (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.