Research map / Vision-language and multimodal / Cross-modal learning
PeVL: Pose-Enhanced Vision-Language Model for Fine-Grained Human Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pre-trained CLIP vision-language model with a pose modality (2D joint coordinates) via cross-modality refinement and semantic-guided multi-level contrastive learning for fine-grained human action recognition.
Problem
Vision-language foundation models do not fully exploit cross-modal information for fine-grained human action recognition because of the large gap between visual and textual modalities; pose may help bridge that gap.
Approach
Extends CLIP with a pose encoder sharing ViT weights, a UCMR block (pose-adapted vision attention and video-enriched pose refinement with refinement supervision), and an SGMC module combining action-level and sub-action-level contrastive associations with a semantic-guided KL loss that weights negatives by text discrepancy.
Key results
- PeVL ViT-L/14 reaches 92.5% Top-1 on Diving48, outperforming AIM ViT-L/14 (90.6%) by 1.9%.
- PeVL ViT-L/14 reaches 97.0% Top-1 on Gym99 and 91.8% on Gym288, surpassing RGBPose-Conv3D (95.6%) on Gym99.
- Ablation on Diving48 shows the full PeVL ViT-B/16 reaches 91.9% vs 87.3% for the simple three-modality design and 88.9% for AIM-like baseline with 809 GFLOPs versus 510 for PeVL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PeVL instead of AIM with ViT-B/16 | Diving48 · Diving48 | Top-1 accuracy: 88.9 -> 91.9 | Helped |
| Use PeVL instead of AIM with ViT-L/14 | Diving48 · Diving48 | Top-1 accuracy: 90.6 -> 92.5 | Helped |
| Add pose modality to the video-text baseline (ablation) | Diving48 · Diving48 | Top-1 accuracy: 60.2 -> 80.3 | Helped |
| Add trainable adapters to the three-modality model (ablation) | Diving48 · Diving48 | Top-1 accuracy: 80.3 -> 87.3 | Helped |
| Add UCMR Block (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 87.3 -> 90.6 | Helped |
| Add SGMC Module (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 90.6 -> 91.9 | Helped |
| Add UCMR Block (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 88.9 -> 91.9 | Helped |
| Add P2V (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 91.1 -> 91.9 | Helped |
| Add V2P (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 90.4 -> 91.9 | Helped |
| Add Refinement Supervision (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 91.0 -> 91.9 | Helped |
| Use all three multi-level contrastive associations instead of only one (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 91.0 -> 91.9 | Helped |
| Replace Semantic Content Guided Loss with vanilla contrastive loss (ablation) | Diving48 PeVL · Diving48 | Top-1 accuracy: 91.9 -> 91.4 | Hurt |
Surprising findings
- Using 2D body joint coordinates as pose input outperforms pose heatmaps by 2.5% Top-1 while requiring fewer GFLOPs (108 vs 311).
Limitations
- Relies on an off-the-shelf pose estimator (HRNet) when datasets lack pose data, so pose errors can propagate.
- Ablations are conducted only on Diving48, not on all four datasets.
Experimental setup
Datasets: Diving48 FineGym HAA500 Toyota-Smarthome Kinetics-400
Benchmarks: Diving48 FineGym Gym99 FineGym Gym288 HAA500 Toyota-Smarthome Kinetics-400
Metrics: Top-1 accuracy Top-5 accuracy mean class accuracy GFLOPs
Baselines: AIM ORViT RGBPose-Conv3D PoseConv3D VT CE TQN EVL MotionFormer Video Swin
Builds on: CLIP AIM PoseConv3D ViLP NT-Xent
Closest papers by meaning
- HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.86)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.86)
- OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction (ICML 2025 · similarity 0.86)
- CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification (NeurIPS 2025 · similarity 0.86)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.