Research map / Vision-language and multimodal / Human and embodied modeling
Understanding Unique Behavioral Patterns through Multimodal Analysis of Eye-Hand Coordination in Autistic Children (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes framing eye-to-hand fixation prediction as sequence-to-sequence machine translation (LSTM/GRU) on VR classroom data to quantify restricted and repetitive behaviors in autistic vs. TD children.
Problem
Existing data-driven ASD behavior analysis mostly uses a single behavioral modality, and multimodal eye-hand data is noisy and irregularly aligned, making the connection between eye movements and hand motions hard to model.
Approach
Treat eye-fixation sequences as the source language and hand-fixation sequences as the target language, training a sequence-to-sequence GRU/LSTM to predict hand fixations from eye fixations, and use the prediction error gap between autistic and TD groups as quantitative evidence of RRBs.
Key results
- Lowest autistic-group test MSE of 0.006265 (GRU-P) vs. TD MSE of 0.01113, with autistic predictions significantly more accurate across all models (P-values as low as 3.9e-16).
- Main task region shows largest autistic-TD gap (MSE 0.005027 vs 0.01015); facilitation region shows no significant difference (P=0.447).
- Autistic prediction error corresponds to about 0.08 units average distance from actual fixation point.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add fixation start and end times to eye fixation positions (ablation) | GRU eye-to-hand prediction · VR eye-to-hand prediction, autistic group | MSE: 0.006265 -> 0.006486 | Hurt |
| Add fixation start and end times to eye fixation positions (ablation) | GRU eye-to-hand prediction · VR eye-to-hand prediction, TD group | MSE: 0.01113 -> 0.01145 | Hurt |
| Use GRU instead of LSTM with position-only inputs (ablation) | Eye-to-hand prediction · VR eye-to-hand prediction, autistic group | MSE: 0.006351 -> 0.006265 | Helped |
| Use GRU instead of LSTM with position-only inputs (ablation) | Eye-to-hand prediction · VR eye-to-hand prediction, TD group | MSE: 0.01204 -> 0.01113 | Helped |
| Use GRU instead of LSTM with position and time inputs (ablation) | Eye-to-hand prediction · VR eye-to-hand prediction, autistic group | MSE: 0.006480 -> 0.006486 | No clear effect |
| Use GRU instead of LSTM with position and time inputs (ablation) | Eye-to-hand prediction · VR eye-to-hand prediction, TD group | MSE: 0.01405 -> 0.01145 | Helped |
| Compare autistic and TD groups | GRU-P eye-to-hand prediction · VR eye-to-hand prediction | MSE: 0.006265 vs 0.01113 | Mixed |
| Compare autistic and TD groups | GRU regional prediction · VR eye-to-hand prediction, main task region | MSE: 0.005027 vs 0.01015 | Helped |
| Compare autistic and TD groups | GRU regional prediction · VR eye-to-hand prediction, facilitation region | MSE: 0.003421 vs 0.004344 | No clear effect |
| Compare autistic and TD groups | GRU regional prediction · VR eye-to-hand prediction, reference region | MSE: 0.002823 vs 0.001813 | Hurt |
Surprising findings
- The reference region shows an opposite trend, with TD children having lower prediction error than autistic children.
Limitations
- Small, specific VR classroom dataset with group-level comparisons only
- Prediction error difference is interpreted as RRB evidence without clinical validation
- Reference region result is unexplained and conflicts with other regions
Experimental setup
Datasets: VR classroom eye and hand fixation data collected via MVCI system
Benchmarks: MVCI VR eye-hand fixation dataset
Metrics: test MSE T-statistic P-value
Baselines: LSTM GRU
Builds on: sequence-to-sequence learning
Closest papers by meaning
- Toward Gaze Target Detection of Young Autistic Children (AAAI 2026 · similarity 0.86)
- AXON: Action Characterization Through Cross-Modal Knowledge Distillation for Neurodiverse Individuals (AAAI 2026 · similarity 0.86)
- Causal Explanation of Quality of Parent-Child Interactions with Multimodal Behavioral Features (Student Abstract) (AAAI 2025 · similarity 0.81)
- EyeGraph: Modularity-aware Spatio Temporal Graph Clustering for Continuous Event-based Eye Tracking (NeurIPS 2024 · similarity 0.81)
- EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics (AAAI 2025 · similarity 0.81)
- Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.