Research map / Label-efficient and robust learning / Semi-supervised learning
SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeFAR, a FixMatch-based semi-supervised framework for fine-grained action recognition that uses dual-level temporal elements, reversal-based moderate temporal perturbation, and uncertainty-based adaptive loss regulation.
Problem
Fine-grained action recognition (FAR) with detailed semantic labels is hard to supervise because fine-grained annotations are costly, and pseudo-labels from teacher models are unstable under semi-supervised training.
Approach
Combines FixMatch-style weak-strong consistency with dual-level temporal sampling (short fine-grained elements plus a long context element), applies frame-order reversal only to fine-grained elements as a strong augmentation, and weights unlabeled losses by teacher prediction confidence and variance across multiple stochastic teacher passes.
Key results
- SeFAR-B reaches 39.0% on Gym99 and 72.8% on FineDiving at 5% labels, outperforming SVFormer (31.4% and 59.1%) on the all-events setting.
- SeFAR-B achieves 50.3% / 77.6% / 87.0% on UCF-101 at 1% / 5% / 10% labels, reporting about 3.3% improvement over prior state of the art on UCF101.
- On Gym-QA, SeFAR features with a frozen encoder and Vicuna-7B reach 49.0 (Gym-QA-99) and 56.2 (Gym-QA-288), above CLIP-ViT-L/16, EVA-CLIP ViT-G/14 and ViT-L/14 features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Dual-Ele to the baseline without the three proposed components (ablation) | SeFAR · Gym99, Gym288, FineDiving | top-1 accuracy: 32.6 -> 34.8; 22.7 -> 25.4; 60.4 -> 64.6 | Helped |
| Add Mod-Perturb to SeFAR with Dual-Ele (ablation) | SeFAR · Gym99, Gym288, FineDiving | top-1 accuracy: 34.8 -> 35.9; 25.4 -> 26.6; 64.6 -> 67.4 | Helped |
| Add Ada-Reg to SeFAR with Dual-Ele and Mod-Perturb (ablation) | SeFAR · Gym99, Gym288, FineDiving | top-1 accuracy: 35.9 -> 36.7; 26.6 -> 27.8; 67.4 -> 72.2 | Helped |
| Use Mod-Perturb instead of spatial-only augmentation (ablation) | SeFAR · Gym99 | top-1 accuracy: 34.2 -> 36.7 | Helped |
| Use Mod-Perturb instead of other temporal perturbations (ablation) | SeFAR · Gym99 | top-1 accuracy: 35.6, 35.2, 36.4, 35.9, 36.0, 36.3 -> 36.7 | Helped |
| Add Ada-Reg (ablation) | SeFAR · FineDiving 1% | top-1 accuracy: 61.5 -> 66.3 | Helped |
| Use SeFAR features instead of the compared pre-trained visual encoders | Vicuna-7B MLLM · Gym-QA-99 | accuracy: 37.3, 43.7, 44.3 -> 49.0 | Helped |
Surprising findings
- Spatial-only augmentation outperforms several temporal augmentations on the Gym-New reversed-action subset, suggesting some temporal perturbations harm direction-sensitive actions.
- Mean-teacher-style teacher predictions on FineGym vary substantially across stochastic passes, motivating the uncertainty-based regulation.
Limitations
- Only temporal augmentation is explored; spatial augmentation is left for future work.
- Relies solely on RGB video input without pose or text modalities.
- Gym-QA evaluation is a constructed multiple-choice setup with limited MLLM comparisons.
Experimental setup
Datasets: FineGym FineDiving UCF101 HMDB51 Something-Something V2
Benchmarks: Gym99 Gym288 FineDiving UCF-101 HMDB-51 Gym-QA-99 Gym-QA-288 Gym-New Something-Something V2
Metrics: top-1 accuracy
Baselines: MemDPC LTG SVFormer MvPL TCLR CMPL TimeBalance FixMatch VideoSSL L2A
Builds on: FixMatch SVFormer TimeSformer Mean Teacher
Closest papers by meaning
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.89)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.88)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.87)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.86)
- Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition (AAAI 2025 · similarity 0.86)
- On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.