Research map / Large language models / AI and society
Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Demonstrates a multimodal (RGBD video, audio, gaze, posture, pointing gesture, object detection) system for detecting nonverbal cues to track common ground and engagement in student group problem-solving.
Problem
Speech-only analysis misses nonverbal cues (pointing, posture, joint attention) needed to judge a student working group's common knowledge and engagement so an AI partner can intervene appropriately.
Approach
Combines Azure Kinect skeleton-based gaze and posture estimation, MediaPipe-based two-stage pointing recognition with pointing-frustum target inference, and a Faster R-CNN object detector to link nonverbal behavior with group knowledge and social-cohesion states in a classroom setting.
Key results
- No quantitative results are reported; the paper is a demo illustrating detection of gaze, posture, pointing, and object states in two classroom scenarios.
Limitations
- No quantitative evaluation of detection accuracy is reported
- Demo uses a simulated classroom with actors and a small 3-participant setup
- Object detection currently targets task-specific objects; portability to other settings is future work
Experimental setup
Datasets: Fibonacci weights task lab videos (annotated object bounding boxes)
Builds on: FasterRCNN
Closest papers by meaning
- MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.85)
- Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.85)
- From Raw Video to Pedagogical Insights: A Unified Framework for Student Behavior Analysis (AAAI 2024 · similarity 0.83)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.83)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.83)
- Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.