Research map / Vision-language and multimodal / Multimodal LLMs

Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CIViL, a multimodal multi-turn customer-support complaint dataset with aspect and severity labels, and VALOR, a two-phase CoT mixture-of-experts with validation experts and a semantic alignment score for fine-grained complaint classification.

Problem

Fine-grained classification of complaint aspects and severity in multi-turn customer support dialogues that include images, which prior work has largely addressed only with unimodal short-form text.

Approach

A two-phase MoE in which CoT experts (DeepSeek-6.7B-based) make predictions from fused BERT/ViT embeddings, followed by validation experts scored with alignment, dominance and complementarity metrics; a meta-fusion MLP combines these with a learned semantic alignment score.

Key results

• VALOR reaches 81.94% aspect accuracy and 72.51% severity accuracy on CIViL, versus 69% and 66% for the best baseline Gemma-3 (9B) per the paper's stated gains.
• Ablation: adding the Validation MoE raises aspect accuracy from 73.74% to 81.94% for the CoT configuration.
• Human win-loss-draw on 200 test samples: win rates of 42.3% (aspect) and 38.5% (severity) for VALOR.

Surprising findings

• Several general-purpose VLM and pretraining baselines reach only about 55-69% accuracy, with some near chance on the six-way aspect task.
• Some ablation variants with validation MoE have lower severity accuracy than the no-validation variants, so the validation module's benefit is not uniform across tasks.

Limitations

• Dataset is small (2,004 conversations) and imbalanced, with Software making up most aspect labels and Packaging only 13 instances.
• Dataset is restricted to Apple Support conversations from one Kaggle corpus, English only.
• Paper reports inconsistent hyperparameters across sections (e.g., learning rate, epochs, batch size) and reports no variance across seeds.

Experimental setup

Datasets: Kaggle Customer Support on Twitter CIViL

Benchmarks: CIViL

Metrics: accuracy macro F1

Baselines: DeepSeek-VL Gemma-3 (9B) Flash Gemini (1.6B) ImageBind Paligemma (3B) SMOL-VLM GIT (300M) FLAVA ALBEF UNITER CLIP ViT-B/32 VisualBERT

Builds on: Mixture-of-Experts chain-of-thought prompting CLIP BERT ViT

Closest papers by meaning

• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators (NeurIPS 2025 · similarity 0.81)
• UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.81)
• Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following (CVPR 2026 · similarity 0.80)
• DEQA: Descriptions Enhanced Question-Answering Framework for Multimodal Aspect-Based Sentiment Analysis (AAAI 2025 · similarity 0.80)
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.79)
• A Framework for Mining Speech-to-Text Transcripts of the Customer for Automated Problem Remediation (AAAI 2024 · similarity 0.79)