Research map / Video / Video segmentation and tracking
VidSole: A Multimodal Dataset for Joint Kinetics Quantification and Disease Detection with Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces tri-axial instrumented insoles and the VidSole multimodal gait dataset (insole, video, motion capture, force plate), plus a GRU-ensemble activity classifier and activity-specific LSTM regressors that estimate knee adduction moment.
Problem
Measuring internal joint loading (knee adduction moment, a knee osteoarthritis risk factor) normally requires expensive lab motion capture and force plates, so there is no low-cost, large-scale way to estimate joint kinetics during daily activities.
Approach
Combine tri-axial force/moment instrumented insoles with pose-estimated knee angles from video, and use a GRU ensemble to classify activities followed by activity-specific LSTM regressors to predict KAM, supported by a new multimodal dataset.
Key results
- Ensemble GRU activity classifier reaches 99.02% test accuracy, versus 97.74% insole-only and 98.28% pose-only.
- Sensor plus knee-angle LSTM walking KAM model reaches r=0.94 and MAE 0.37 %BW*ht, below the 0.5 %BW*ht clinical threshold.
- Across the four activities, KAM correlation coefficients range from 0.64 to 0.94 with the multimodal inputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine video and insole predictions instead of using either input alone (ablation) | GRU activity classifier · Activity classification test set | accuracy: 97.74% / 98.28% -> 99.02% | Helped |
| Add pose-derived knee angle to insole sensor inputs (ablation) | LSTM KAM regression · Walking | MAE (%BW*ht): 0.42±0.18 -> 0.37±0.16 | Helped |
| Add pose-derived knee angle to insole sensor inputs (ablation) | LSTM KAM regression · Running | MAE (%BW*ht): 0.55±0.28 -> 0.46±0.16 | Helped |
| Add pose-derived knee angle to insole sensor inputs (ablation) | LSTM KAM regression · Stand-to-Sit | r: 0.40±0.52 -> 0.64±0.28 | Helped |
| Add pose-derived knee angle to insole sensor inputs (ablation) | LSTM KAM regression · Sit-to-Stand | r: 0.54±0.42 -> 0.64±0.40 | Helped |
| Add pose-derived knee angle to insole sensor inputs (ablation) | LSTM KAM regression · Walking | r: 0.94±0.05 -> 0.94±0.04 | No clear effect |
Surprising findings
- Sit-to-stand and stand-to-sit KAM correlations were weak (r around 0.40-0.54) with insole data alone, improving to about 0.64 only after adding knee angle.
Limitations
- Data collected in a gait lab with healthy young participants (ages 23.4 ± 4.0), not in clinical or real-world settings.
- Not validated on individuals with knee osteoarthritis or older adults.
- Sit-to-stand and stand-to-sit KAM estimates are notably less accurate than walking and running.
Experimental setup
Datasets: VidSole
Benchmarks: VidSole
Metrics: accuracy mean absolute error Pearson correlation coefficient
Baselines: OpenCap motion-capture-only KAM estimation insole-only model pose-based video model
Builds on: GRU LSTM MediaPipe ADAM
Closest papers by meaning
- MosaicMRI: A Diverse Dataset and Benchmark for Raw Musculoskeletal MRI (NeurIPS 2026 · similarity 0.84)
- Muscles in Time: Learning to Understand Human Motion In-Depth by Simulating Muscle Activations (NeurIPS 2024 · similarity 0.83)
- ReMAP: Neural Model Reprogramming with Network Inversion and Retrieval-Augmented Mapping for Adaptive Motion Forecasting (NeurIPS 2024 · similarity 0.83)
- SkeletonGait: Gait Recognition Using Skeleton Maps (AAAI 2024 · similarity 0.82)
- HIT: Estimating Internal Human Implicit Tissues from the Body Surface (CVPR 2024 · similarity 0.82)
- Care-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson’s Disease Gait Assessment (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.