Research map / Vision-language and multimodal / Cross-modal learning
Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions
Ting Han, Casey Kennington, David Schlangen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace rule-based utterance segmenter with LSTM | utterance segmenter · Utterance segmentation | accuracy: 0.85 -> 0.92 | Improved |
| Replace rule-based utterance segmenter with LSTM | utterance segmenter · Utterance segmentation | F1: 0.84 -> 0.89 | Improved |
| Replace rule-based utterance segmenter with LSTM | utterance segmenter · Utterance segmentation | std of accuracy across participants: 0.23 -> 0.10 | Improved |
| Use Gaussian KDE gesture interpretation instead of chance | gesture interpretation · Object reference | accuracy: 1/3 -> 0.81 | Improved |
| Use full system with speech only instead of random baseline | Our System · Scene retrieval | accuracy: 0.17 -> 0.75 | Improved |
| Use gesture only (language weight 0) instead of random baseline | Our System · Scene retrieval | accuracy: 0.17 -> 0.50 | Improved |
| Add gestures to speech in fusion | Our System · Scene retrieval | accuracy: 0.75 -> 0.84 | Improved |
| Add gestures to speech in fusion | Our System · Scene retrieval | MRR: 0.79 -> 0.84 | Improved |
| Add gestures to speech for human crowd workers | human evaluation · Scene retrieval | accuracy: 0.86 -> 0.77 | Hurt |
| Add gestures to speech in incremental decision | Our System · Incremental eval | average fc: 0.24 -> 0.24 | No clear effect |
| Add gestures to speech in incremental decision | Our System · Incremental eval | average ff: 0.67 -> 0.65 | Improved |
| Add gestures to speech in incremental decision | Our System · Incremental eval | average eo (edit overhead) | Hurt |
Closest papers by meaning
- Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities (ICLR 2025 · similarity 0.87)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
- Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.86)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.86)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.86)
- SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
