Research map / Vision-language and multimodal / Multimodal LLMs

Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VRSLU, a profile-based SLU benchmark with GPT-4o/FLUX.1-dev-generated scene images replacing one-hot context awareness plus human-refined reasoning, and LR-Instruct, which predicts labels before generating reasoning.

Problem

Existing profile-based SLU datasets represent context awareness as idealized one-hot vectors and provide no reasoning annotations, so they do not reflect real-world deployment conditions.

Approach

Replace one-hot context vectors with generated scene images (verified by humans), add human-verified reasoning explanations for intent and slot labels, and train MLLMs with an instruction template that predicts labels first and reasoning second to limit reasoning errors propagating into labels.

Key results

• LR-Instruct on Qwen2.5-VL-7B raises overall accuracy from 82.86 to 86.63 on VRSLU.
• Qwen2-VL-2B with LR-Instruct improves overall accuracy from 69.11 to 72.76.
• Removing the verified CA image drops Qwen2.5-VL-7B+LR-Instruct overall accuracy by 3.39 points (86.63 to 83.24).

Surprising findings

• Explicit reasoning generation before label prediction lowered Qwen2.5-VL-7B performance relative to the no-reasoning setting, showing flawed reasoning can hurt labels.
• Directly generating CA images from one-hot vectors with FLUX.1-dev gave unsatisfactory results, requiring a GPT-4o description step.

Limitations

• Scene images are synthetic rather than captured from real users, and only manually verified by two annotators.
• Reasoning relies on GPT-4o generation with human correction, so reasoning quality depends on GPT-4o and annotator effort.
• Evaluation covers only a small set of Qwen/Yi MLLMs fine-tuned with LoRA on one dataset.

Experimental setup

Datasets: ProSLU

Benchmarks: VRSLU

Metrics: intent accuracy slot F1 overall accuracy BLEU-1 BLEU-4 cosine similarity

Baselines: Yi-VL-6B Qwen2-VL-2B Qwen2-VL-7B Qwen2.5-VL-3B Qwen2.5-VL-7B General SLU ProHAN

Builds on: ProSLU LoRA GPT-4o

Closest papers by meaning

• Uni-MIS: United Multiple Intent Spoken Language Understanding via Multi-View Intent-Slot Interaction (AAAI 2024 · similarity 0.88)
• Towards Multi-Intent Spoken Language Understanding via Hierarchical Attention and Optimal Transport (AAAI 2024 · similarity 0.88)
• Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling via Adjustive and Forced Cross-Task Alignment (AAAI 2024 · similarity 0.88)
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.87)
• VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridging (CVPR 2026 · similarity 0.86)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.86)