Research map / Video / Video segmentation and tracking
LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs ~50 hours of monocular table tennis broadcast video into 3D exchanges (73,222) and trains an uncertainty-aware transformer with conformal prediction to pre-position a simulated KUKA arm, raising ball return rate from 49.9% to 59.0%.
Problem
Robots struggle against professional table tennis players in fast rallies because they cannot anticipate opponent shots, and prior anticipation work is limited by small in-house datasets.
Approach
Scalable pipeline that reconstructs 3D player SMPL meshes and ball trajectories from monocular online competition videos (camera calibration from table corners, parabolic bounce detection, projectile fit with Stokes drag), then trains a transformer on the resulting dataset to forecast opponent shots with ensemble-based conformal prediction regions used to pre-position a robot.
Key results
- Anticipatory pre-positioning raises ball return rate from 49.9% (baseline) to 59.0% against high-speed hits in simulation; oracle reaches 64.5%.
- Empirical conformal coverage closely matches theory, e.g. 0.885 per-axis coverage for target 0.90 at alpha=0.10.
- Reconstruction yields 73,222 exchanges; estimated ball position error about 8.9 cm and player joint error about 28 cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use anticipatory predictions to pre-position the robot before the opponent hits | KUKA robotic arm table-tennis controller · Simulation held-out test exchanges | ball return rate: 49.9% -> 59.0% | Helped |
| Use anticipatory predictions to pre-position the robot before the opponent hits | KUKA robotic arm table-tennis controller · Simulation held-out test exchanges | return accuracy (mean deviation from target bounce point): 0.497 m -> 0.463 m | Helped |
| Use anticipatory predictions to pre-position the robot before the opponent hits | KUKA robotic arm table-tennis controller · Simulation held-out test exchanges | pose accuracy (position error / orientation error): 0.25 m / 13.3◦ -> 0.19 m / 9.86◦ | Helped |
Surprising findings
- Conformal regions usually over-cover the table on extreme hits, though biased regions point toward the correct side more than 85% of the time.
Limitations
- Reconstruction is approximate (about 8.9 cm ball and 28 cm player joint error) and racket poses are not recovered.
- Dataset is biased: last winning segments are often missed and players tend to hit to their right.
- Results are only in simulation; validation on physical hardware is future work.
Experimental setup
Datasets: online table tennis competition recordings (~800 h raw, ~50 h gameplay) 10,000 manually labeled ball frames for TrackNetV3 fine-tuning
Benchmarks: held-out table tennis test set of 1,000 exchanges (simulated KUKA arm in MuJoCo)
Metrics: ball return rate return accuracy pose accuracy empirical conformal coverage reprojection error
Baselines: Non-anticipatory baseline controller Oracle pre-positioning
Builds on: HMR 2.0 TrackNetV3 YOLOv8 conformal prediction IDDM
Closest papers by meaning
- ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis (AAAI 2024 · similarity 0.84)
- Event2Tracking: Reconstructing Multi-Agent Soccer Trajectories Using Long-Term Multimodal Context (AAAI 2025 · similarity 0.83)
- The CoachAI Badminton Environment: A Novel Reinforcement Learning Environment with Realistic Opponents (Student Abstract) (AAAI 2024 · similarity 0.83)
- The CoachAI Badminton Environment: Bridging the Gap between a Reinforcement Learning Environment and Real-World Badminton Games (AAAI 2024 · similarity 0.83)
- SkillMimic: Learning Basketball Interaction Skills from Demonstrations (CVPR 2025 · similarity 0.83)
- The Indoor-Training Effect: Unexpected Gains from Distribution Shifts in the Transition Function (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.