Research map / Vision-language and multimodal / Visual question answering
mmWalk: Towards Multi-modal Multi-view Walking Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds mmWalk, a CARLA-simulated multi-view (walker, guide dog, drone), multi-modal (RGB, depth, semantic) walking dataset with BLV corner cases and landmarks, plus mmWalkVQA (69k GPT-4o-generated VQA triplets), benchmarks VLMs zero/3-shot, and LoRA-finetunes InternVL2-8B showing gains including on EgoTextVQA.
Problem
Walking assistance for blind and low-vision (BLV) people in complex outdoor environments lacks holistic, safety-aware scene understanding and suitable multi-view multi-modal data.
Approach
A simulated multi-view, panoramic, multi-modal walking dataset annotated with BLV-specific corner cases and ATmaps-derived landmarks, with a VQA benchmark generated by GPT-4o from clockwise BLV-friendly spatial descriptions of depth and semantic data.
Key results
- Finetuned InternVL2-8B reaches an average normalized score of 55.21 on mmWalkVQA, up 13.86 points over its zero-shot score of 41.35.
- Zero-shot InternVL2-8B is the best zero-shot model (41.35 average), and LLaVA-NeXT-7B is the top 3-shot model (43.71 average).
- The mmWalk-finetuned InternVL2-8B improves overall EgoTextVQA frame-input score to 21.55/1.92 from 18.5/1.4 for the original InternVL2-8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune InternVL2-8B on mmWalkVQA | InternVL2-8B · mmWalkVQA | normalized average score: 41.35 -> 55.21; 13.86% | Helped |
| provide 3-shot examples | LLaVA-NEXT-7B · mmWalkVQA | normalized average score: 39.84 -> 43.71; 3.87 | Helped |
| provide 3-shot examples | Qwen2VL-7B-Instruct · mmWalkVQA | normalized average score: 37.91 -> 41.89; 3.98 | Helped |
| provide 3-shot examples | Janus-Pro-7B · mmWalkVQA | normalized average score: 37.82 -> 37.75; -0.07 | Hurt |
| provide 3-shot examples | Chameleon-7B · mmWalkVQA | normalized average score: 21.14 -> 22.48; 1.34 | Helped |
| fine-tune InternVL2-8B on mmWalk and evaluate with frames | InternVL2-8B · EgoTextVQA | Overall score: 18.5/1.4 -> 21.55/1.92 | Helped |
| replace full walker, dog, and drone input with walker view only (ablation) | InternVL2-8B (fine-tuned) · mmWalkVQA, average without M3 | normalized average score: 55.05 -> 53.19; -1.86 | Hurt |
| replace full walker, dog, and drone input with walker and dog views (ablation) | InternVL2-8B (fine-tuned) · mmWalkVQA, average without M3 | normalized average score: 55.05 -> 53.85; -1.20 | Hurt |
| replace full walker, dog, and drone input with walker and drone views (ablation) | InternVL2-8B (fine-tuned) · mmWalkVQA, average without M3 | normalized average score: 55.05 -> 54.08; -0.97 | Hurt |
| replace full walker, dog, and drone input with walker view only (ablation) | LLaVA-NEXT-7B (zero-shot) · mmWalkVQA, average without M3 | normalized average score: 41.20 -> 39.59; -1.61 | Hurt |
| replace full walker, dog, and drone input with walker and dog views (ablation) | LLaVA-NEXT-7B (zero-shot) · mmWalkVQA, average without M3 | normalized average score: 41.20 -> 36.06; -5.14 | Hurt |
| replace full walker, dog, and drone input with walker and drone views (ablation) | LLaVA-NEXT-7B (zero-shot) · mmWalkVQA, average without M3 | normalized average score: 41.20 -> 38.58; -2.62 | Hurt |
Surprising findings
- Drone-only or single-view inputs are sometimes better than full multi-view input on specific categories, and the drone view is least effective for risk assessment (H1).
- Spatial reasoning (M1) is the hardest category for both zero-shot and 3-shot models.
Limitations
- Dataset is purely synthetic from CARLA and may not capture real-world BLV diversity.
- LLM-as-a-judge evaluation may carry biases.
- IMU data, sequential frames, and semantic labels are not fully exploited.
Experimental setup
Datasets: mmWalk mmWalkVQA EgoTextVQA
Benchmarks: mmWalkVQA EgoTextVQA
Metrics: normalized LLM-judge score (0-100) Spearman correlation with human ratings
Baselines: Chameleon-7B Janus-Pro-7B Qwen2VL-7B-Instruct LLaVA-NeXT-7B LLaVA-Onevision-7B InternVL2-8B
Builds on: CARLA LLaVA-NeXT InternVL2 EgoTextVQA
Closest papers by meaning
- WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation (CVPR 2026 · similarity 0.86)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.85)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.85)
- Text to Blind Motion (NeurIPS 2024 · similarity 0.84)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.84)
- EgoBlind: Towards Egocentric Visual Assistance for the Blind (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.