Research map / Vision-language and multimodal / Multimodal LLMs
Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Period-LLM is a LLaVA-style multimodal LLM trained with an easy-to-hard curriculum (text repetition QA, then Countix-QA and rPPG-QA video tasks) plus a gradient-reweighting "Resisting Logical Oblivion" strategy, evaluated on new periodic QA benchmarks.
Problem
Current MLLMs struggle with periodic tasks such as counting repetitive motions and estimating physiological periodicity (e.g., heart rate), due to lack of temporal modeling, conflict between long-period reasoning and short-term semantic alignment, and scarce count-supervised captions.
Approach
An easy-to-hard curriculum that pretrains on repeated-word text QA before video periodic QA, combined with a channel-wise gradient reweighting (RLO) that boosts updates to under-updated feature channels to avoid forgetting periodic reasoning during semantic alignment.
Key results
- On Countix-QA fine-tuned, Period-LLM reaches MAE 3.77 and CIDEr 0.810 versus the best baseline MAE around 4.6-5.0 (e.g., Video-ChatGPT 4.64).
- On rPPG-QA fine-tuned, MAE drops to 13.78 from 15.45 without easy-to-hard training and RLO (ablation).
- The beta=0.05 setting in the RLO weight function gives the lowest Countix-QA MAE (3.77); mean threshold beats median (3.85) and learnable (4.20).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on text-QA before Countix-QA fine-tuning (ablation) | Period-LLM · Countix-QA | MAE: 4.30 -> 3.89 | Helped |
| Add RLO after text-QA and Countix-QA training (ablation) | Period-LLM · Countix-QA | MAE: 3.89 -> 3.77 | Helped |
| Add RLO after text-QA and Countix-QA training (ablation) | Period-LLM · Countix-QA | CIDEr: 0.782 -> 0.810 | Helped |
| Pretrain on text-QA before rPPG-QA training (ablation) | Period-LLM · rPPG-QA | MAE: 15.45 -> 14.46 | Helped |
| Add Countix-QA pretraining after text-QA pretraining (ablation) | Period-LLM · rPPG-QA | MAE: 14.46 -> 14.05 | Helped |
| Add RLO after text-QA, Countix-QA, and rPPG-QA training (ablation) | Period-LLM · rPPG-QA | MAE: 14.05 -> 13.78 | Helped |
| Set β to 0.01 instead of the default 0.05 (ablation) | Period-LLM RLO · Countix-QA | MAE: 3.77 -> 3.85 | Hurt |
| Set β to 0.1 instead of the default 0.05 (ablation) | Period-LLM RLO · Countix-QA | MAE: 3.77 -> 3.90 | Hurt |
| Set β to 0.5 instead of the default 0.05 (ablation) | Period-LLM RLO · Countix-QA | MAE: 3.77 -> 4.05 | Hurt |
| Replace the mean threshold with the median threshold (ablation) | Period-LLM RLO · Countix-QA | MAE: 3.77 -> 3.85 | Hurt |
| Replace the mean threshold with a learnable threshold (ablation) | Period-LLM RLO · Countix-QA | MAE: 3.77 -> 4.20 | Hurt |
| Use Period-LLM instead of VideoLLaMA | LLaMA-7B · Countix-QA zero-shot | MAE: 6.18 -> 5.51 | Helped |
Surprising findings
- A learnable threshold for the RLO weighting performs worse than fixed mean or median thresholds, suggesting instability.
- On the Drive-QA dataset Period-LLM's RMSE (105.54) is not clearly the best in the zero-shot setting for some baselines, with mixed results across metrics.
Limitations
- Paper gives vague descriptions of the RLO mechanism and the threshold/weight computation, limiting reproducibility.
- Evaluation relies on self-constructed QA datasets derived from existing corpora with GPT-4-generated text, and baselines are reported with unclear fine-tuning setups.
- Results on non-visual modalities (Drive-QA, Radar-QA) are mixed and lack detailed analysis.
Experimental setup
Datasets: Countix Countix-QA rPPG-QA VIPL-HR PURE UBFC-rPPG V4V BUAA-MIHR MNIST RotNIST-QA T_DRIVE20150206 Drive-QA
Benchmarks: Countix-QA rPPG-QA RotNIST-QA Drive-QA Radar-QA
Metrics: MAE RMSE BLEU-1 CIDEr METEOR
Baselines: FrozenBiLM VideoLLaMA LLaMA-Adapter VideoChat Video-ChatGPT BT-Adapter LLaMA-VID
Builds on: LLaVA CLIP Countix
Closest papers by meaning
- Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling (NeurIPS 2025 · similarity 0.88)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.87)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
- LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.