Research map / Video / Action recognition
From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a VerbNet-based structured action semantic space and a unified multi-modal database (Pangea, 28 datasets) and trains a physical-to-semantic (P2S) mapping model with disentangled node representations and hyperbolic alignment for action recognition and transfer.
Problem
Action datasets use incompatible, idiosyncratic class definitions with semantic gaps and mismatched granularity ("isolated islands"), which hinders integration, cross-dataset learning, and few/zero-shot transfer.
Approach
Use the VerbNet verb hierarchy as a structured semantic space, align classes of many datasets to its nodes to form a unified database, and map physical inputs (image/video/3D human) to multiple verb nodes via disentangled node-specific representations and Lorentz hyperbolic embeddings with entailment loss.
Key results
- On the Pangea verb-node test set, P2S-aug reaches 34.25 mAP versus 28.25 for CLIP.
- P2S + EVL reaches 80.87% top-1 on HAA and 85.09% on HMDB51, improving over EVL alone.
- P2S achieves 54.1% top-1 on HAA4D, above SGN (53.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add label augmentation (ablation) | P2S · Pangea test set | mAP: 34.01 -> 34.25 | Helped |
| Use P2S instead of CLIP | HICO · HICO | mAP: 46.35 -> 47.74 | Helped |
| Fuse P2S with EVL | HAA · HAA | Top-1 Accuracy (%): 76.40 -> 80.87 | Helped |
| Fuse P2S with EVL | HMDB51 · HMDB51 | Top-1 Accuracy (%): 83.68 -> 85.09 | Helped |
| Fuse P2S with EVL | Kinetics-400 · Kinetics-400 | Top-1 Acc (%): 87.64 -> 90.22 | Helped |
| Use P2S instead of CLIP-Pangea | BABEL-120 · BABEL-120 | Top-1%: 48.53 -> 49.69 | Helped |
| Use P2S instead of SGN | HAA4D · HAA4D | Top-1%: 53.3 -> 54.1 | Helped |
| Add semantic disentanglement (ablation) | P2S-aug · Pangea test set | mAP: 30.09 -> 34.25 | Helped |
| Add semantic augmentation (ablation) | P2S · Pangea test set | mAP: 34.01 -> 34.25 | Helped |
| Add the text encoder (ablation) | P2S-aug · Pangea test set | mAP: 31.81 -> 34.25 | Helped |
| Add hyperbolic mapping (ablation) | P2S-aug · Pangea test set | mAP: 32.56 -> 34.25 | Helped |
| Add semantic prompt (ablation) | P2S-aug · Pangea test set | mAP: 33.20 -> 34.25 | Helped |
Surprising findings
- CLIP trained on the mixed Pangea data (CLIP-Pangea) performs worse than vanilla CLIP on HICO and HMDB51 (e.g., 67.47 to 67.69 on HMDB51 is slightly higher, but 72.82 to 70.45 top-1 on Kinetics-400), indicating naive mixing of datasets can hurt.
Limitations
- Pseudo 3D labels from 2D data via ROMP and EFT are noisy.
- Uses relatively concise models; larger models trained on Pangea are left for future work.
- Semantic space built from VerbNet verbs only; mapping of classes relies partly on GPT-3.5 and manual annotation.
Experimental setup
Datasets: Pangea HICO HAA HMDB51 Kinetics-400 BABEL HAA4D VerbNet HAA500 ActivityNet NTU RGB+D Human3.6M
Benchmarks: Pangea HICO HAA HMDB51 Kinetics-400 BABEL-120 HAA4D
Metrics: mAP top-1 accuracy top-5 accuracy
Baselines: CLIP CLIP-Pangea PointNet++ PointNet++-Pangea RelViT EVL TSN VideoMAE SGN 2s-AGCN
Builds on: CLIP VerbNet PointNet++ Lorentz hyperbolic embeddings LLaMA-Adapter V2
Closest papers by meaning
- Open-Vocabulary Video Relation Extraction (AAAI 2024 · similarity 0.86)
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.86)
- On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding (ICLR 2026 · similarity 0.86)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.85)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.85)
- MA-Bench: Towards Fine-grained Micro-Action Understanding (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.