Research map / 3D vision / 3D generation
Learning from Synthetic Human Group Activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces M3Act, a Unity-based synthetic data generator for multi-view, multi-group, multi-person human actions and group activities, and shows it improves multi-person tracking (MOTRv2 on DanceTrack), group activity recognition, and enables a new controllable 3D group activity generation task.
Problem
Real-world datasets for human group activity understanding are costly to collect and annotate, and existing synthetic human datasets focus on independently animated persons rather than coordinated single- and multi-group activities.
Approach
A procedural Unity-based generator that authors parameterized group activities with multiple semantic groups, domain randomization, and rich 2D/3D, tracking, and group/atomic action annotations, used both to pre-train/mix-train existing recognition and tracking models and to create a new 3D group activity generation benchmark.
Key results
- Adding M3Act synthetic dance and walk/run data to DanceTrack training raises MOTRv2 HOTA from 68.8 to 71.9, moving the DanceTrack leaderboard rank from 10th to 2nd.
- Pre-training Composer on 100% of synthetic data improves CAD2 group accuracy by 4.87 points and person accuracy by 7.43 points.
- MDM+IFormer lowers collision frequency from 3.643 (MDM) to 1.157 in generated 3D group activities, with FID 3.242 vs 3.909.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add M3Act walk and run groups to MOTRv2 training data (ablation) | MOTRv2 · DanceTrack | HOTA: 68.8 -> 70.1 | Helped |
| Add M3Act dance groups to MOTRv2 training data (ablation) | MOTRv2 · DanceTrack | HOTA: 68.8 -> 59.0 | Hurt |
| Add M3Act walk and run groups plus dance groups to MOTRv2 training data (ablation) | MOTRv2 · DanceTrack | HOTA: 68.8 -> 71.9 | Helped |
| Pretrain Composer on 100% rather than 0% M3Act synthetic data (ablation) | Composer · CAD2 | Group activity accuracy: 84.87±2.3 -> 89.74±1.0 | Helped |
| Pretrain Composer on 100% rather than 0% M3Act synthetic data (ablation) | Composer · CAD2 | Person action accuracy: 81.31±2.4 -> 88.74±1.7 | Helped |
| Pretrain Actor Transformer on 100% rather than 0% M3Act synthetic data (ablation) | Actor Transformer · CAD2 | Group activity accuracy: 78.08±1.0 -> 83.67±1.2 | Helped |
| Pretrain Actor Transformer on 100% rather than 0% M3Act synthetic data (ablation) | Actor Transformer · CAD2 | Person action accuracy: 76.22±2.2 -> 81.65±1.2 | Helped |
| Increase M3Act pretraining data from 10% to 100% (ablation) | Composer · CAD2 | Group activity accuracy: 86.12±1.8 -> 89.74±1.0 | Helped |
| Increase M3Act pretraining data from 10% to 100% (ablation) | Actor Transformer · CAD2 | Group activity accuracy: 77.59±2.4 -> 83.67±1.2 | Helped |
| Add an interaction transformer across persons to MDM (ablation) | MDM · 3D group activity generation test split | Group-level FID: 3.909 ± 0.019 -> 3.242 ± 0.016 | Helped |
| Add an interaction transformer across persons to MDM (ablation) | MDM · 3D group activity generation test split | Collision frequency: 3.643 -> 1.157 | Helped |
Surprising findings
- Adding walk/run synthetic groups helped DanceTrack more than adding the dance groups, despite DanceTrack being a dance-focused dataset.
- Synthetic data from BEDLAM and GTA-Humans reduced DanceTrack performance substantially when mixed in (HOTA about 55 and 54 vs 68.8 baseline).
Limitations
- Domain gap between synthetic and real-world data remains, limited by the asset set.
- Group behaviors are constrained by the heuristics used for activity authoring.
- The generative GAG baseline still underperforms the procedural ground truth.
Experimental setup
Datasets: DanceTrack CAD2 Volleyball Dataset M3ActRGB M3Act3D BEDLAM GTA-Humans
Benchmarks: DanceTrack CAD2 Volleyball Dataset
Metrics: HOTA DetA AssA IDF1 MOTA top-1 accuracy FID diversity multimodality collision frequency
Baselines: MOTRv2 MOTR MeMOTR CO-MOT Composer Actor Transformer MDM BEDLAM GTA-Humans
Builds on: MOTRv2 MDM Composer Actor Transformer PeopleSansPeople
Closest papers by meaning
- Learning Group Actions on Latent Representations (NeurIPS 2024 · similarity 0.86)
- MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human Captures (CVPR 2024 · similarity 0.85)
- M3GYM: A Large-Scale Multimodal Multi-view Multi-person Pose Dataset for Fitness Activity Understanding in Real-world Settings (CVPR 2025 · similarity 0.85)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.84)
- Motions as Queries: One-Stage Multi-Person Holistic Human Motion Capture (CVPR 2025 · similarity 0.84)
- ReGenNet: Towards Human Action-Reaction Synthesis (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.