Research map / Vision-language and multimodal / Multimodal LLMs

HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HandVQA, a 1.6M-question multiple-choice benchmark built from 3D hand datasets to diagnose VLM spatial reasoning about hand joint angles, distances and relative positions, and shows LoRA fine-tuning on it transfers zero-shot to gesture and hand-object interaction recognition.

Problem

Current vision-language models struggle with fine-grained spatial reasoning about articulated hand poses (joint angles, distances, relative positions), and it is unclear whether this can be diagnosed and improved with controlled VQA data.

Approach

Deterministically converts normalized 3D hand joint coordinates from FreiHAND, InterHand2.6M and FPHA into templated multiple-choice questions over discretized angle, distance and axis-wise relative-position descriptors, then fine-tunes VLMs with LoRA and tests transfer to unseen hand tasks.

Key results

• Base VLMs perform near or below chance on many descriptors; after LoRA fine-tuning, relative position accuracy reaches about 97% for LLaVA on InterHand2.6M, while angle accuracy stays below 75% (best 74.35%).
• Fine-tuned Qwen 2.5 VL raises HaGRID gesture recognition accuracy from 71.86% to 82.19%, and LLaVA from 57.42% to 69.58%.
• Fine-tuned Qwen 2.5 VL improves H2O hand-object interaction recognition from 80.26% to 82.89%.

Surprising findings

• Base VLMs often answer 'close' for distances and 'bent slightly inward' for angles regardless of the image, indicating superficial answering.
• Fine-tuned models still struggle with joint angles even though relative position accuracy exceeds 90%.

Limitations

• Only 7B models evaluated due to GPU constraints
• Angle and distance thresholds are fixed and discretized; aligned cases removed
• LoRA-only fine-tuning with frozen vision encoder limits angle accuracy

Experimental setup

Datasets: FreiHAND InterHand2.6M FPHA HaGRID H2O

Benchmarks: HandVQA HaGRID gesture MCQ H2O hand-object interaction MCQ

Metrics: accuracy mean absolute error

Baselines: LLaVA Mistral 7B DeepSeek Janus Pro 7B Qwen 2.5 VL 7B Instruct

Builds on: LoRA LLaVA Qwen2.5-VL Janus Pro

Closest papers by meaning

• GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.88)
• How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.87)
• BOP-ASK: Object-Interaction Reasoning for Vision-Language Models (CVPR 2026 · similarity 0.87)
• HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models (CVPR 2025 · similarity 0.86)
• Hand1000: Generating Realistic Hands from Text with Only 1,000 Images (AAAI 2025 · similarity 0.86)
• SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.86)