Research map / 3D vision / 3D generation
LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LucidAction, a multi-view, multi-modal MoCap-based action quality assessment dataset of 8 gymnastics events with four curriculum levels and penalty-based scoring, and benchmarks contrastive regression baselines on it.
Problem
Action quality assessment (AQA) is limited by small, mono-modal datasets drawn from one-shot web competitions, lacking pose data, step-wise scores, and progressive skill-level coverage.
Approach
Organize AQA data along a curriculum-learning hierarchy (sports event, curriculum level, action) captured with an 8-view 4K MoCap system, add 2D/3D pose and penalty-item-based scores, and use contrastive regression baselines with auxiliary penalty heads to show the value of multi-modal and fine-grained annotations.
Key results
- On LucidAction, TPT reaches Spearman 0.701 and R-l2 0.624 versus USDL's 0.540 and 0.708 on RGB video.
- Multi-view channel concatenation after the pair encoder raises Spearman from 0.701 to 0.759 (Mv.Test).
- Adding RGB plus 3D pose dual-stream input raises Spearman to 0.746; adding penalty heads raises it to 0.741 with R-l2 0.501.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use contrastive regression (TPT) instead of direct regression (USDL) | AQA models · LucidAction | Spearman’s rank correlation (ρ): 0.540 -> 0.701 | Helped |
| Use contrastive regression (TPT) instead of direct regression (USDL) | AQA models · LucidAction | R-ℓ2(×100): 0.708 -> 0.624 | Helped |
| Add multi-view inputs as separate batch samples (ablation) | TPT · LucidAction | Spearman’s rank correlation (ρ): 0.701 -> 0.730 | Helped |
| Concatenate multi-view channels after the pair encoder (ablation) | TPT · LucidAction, Sv.Test | Spearman’s rank correlation (ρ): 0.701 -> 0.747 | Helped |
| Use 2D pose instead of RGB input (ablation) | TPT · LucidAction | Spearman’s rank correlation (ρ): 0.701 -> 0.605 | Hurt |
| Use 3D pose instead of RGB input (ablation) | TPT · LucidAction | R-ℓ2(×100): 0.624 -> 0.593 | Helped |
| Combine RGB and 3D pose in a dual stream (ablation) | TPT · LucidAction | Spearman’s rank correlation (ρ): 0.701 -> 0.746 | Helped |
| Add penalty-head supervision (ablation) | TPT · LucidAction | Spearman’s rank correlation (ρ): 0.701 -> 0.741 | Helped |
| Add penalty-head supervision (ablation) | TPT · LucidAction | R-ℓ2(×100): 0.624 -> 0.501 | Helped |
Surprising findings
- 2D pose alone degrades performance (Spearman 0.605 vs 0.701 for RGB), while 3D pose alone is competitive in R-l2.
- Single-view test with multi-view training yields gains close to multi-view testing for some fusion settings (e.g., 0.747 Sv.Test vs 0.759 Mv.Test).
Limitations
- Data collected in controlled lab environments, may not reflect real-world lighting and backgrounds.
- Subjective bias may remain in penalty-based scoring by annotators.
- 3D poses are reconstructed from multi-view 2D pose estimates used as ground truth rather than independent mocap labels.
Experimental setup
Datasets: LucidAction AQA-7 MTL-AQA FineDiving NTU RGB+D
Benchmarks: LucidAction AQA-7 MTL-AQA FineDiving
Metrics: Spearman's rank correlation relative L2 distance
Baselines: USDL CoRe TSA TPT Siamese Network
Builds on: TPT CoRe TSA USDL RTMPose
Closest papers by meaning
- FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.86)
- Learning Skill-Attributes for Transferable Assessment in Video (NeurIPS 2025 · similarity 0.86)
- GAIA: Rethinking Action Quality Assessment for AI-Generated Videos (NeurIPS 2024 · similarity 0.86)
- ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.85)
- ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis (AAAI 2024 · similarity 0.85)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.