Research map / Vision-language and multimodal / Visual question answering

DEQA: Descriptions Enhanced Question-Answering Framework for Multimodal Aspect-Based Sentiment Analysis

AAAI 2025 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DEQA, which uses GPT-4-generated image descriptions and frames MABSA as multi-turn QA with three-expert (text, text+description, text+vision) decision ensembles for MATE and MASC.

Problem

Multimodal aspect-based sentiment analysis (MATE and MASC on image-text tweets) suffers from the gap between image and text modalities, which modality-alignment pre-training methods address only indirectly and at high computational cost.

Approach

Generate textual image descriptions with a multimodal LLM (GPT-4 vision) as a direct bridge to text, cast MABSA as multi-turn extraction/validation/classification queries, and fuse outputs of text-only, text+description, and text+vision experts via decision ensembles, without pre-training.

Key results

• Achieves 88.0 MATE F1 on Twitter2015 and 94.4 on Twitter2017, and 72.7 MABSA Micro-F1 on Twitter2015 and 71.9 on Twitter2017, surpassing prior SOTA on both datasets for MABSA.
• On MASC, DEQA reaches 82.1 accuracy on Twitter2015 (vs. AoM 80.2), but on Twitter2017 it is only marginally better in Macro-F1 (75.1 vs 75.0) and behind AoM in accuracy (75.8 vs 76.4).
• GPT-4o achieves only 25.9 (Twitter2015) and 25.7 (Twitter2017) MABSA Micro-F1, far below DEQA's 72.7 and 71.9.

Measured effects reported in the paper

Surprising findings

• Removing the image while keeping the GPT-4 description causes a larger performance drop on MABSA than removing the description while keeping the image, which the authors interpret as descriptions being more important than images.
• Zero-shot GPT-4o performs very poorly on MABSA (about 26 F1), far below fine-tuned models.

Limitations

• On Twitter2017 MASC, gains over AoM are negligible or negative in accuracy, which the authors attribute partly to unknown emojis for DeBERTa.
• Relies on GPT-4 vision API to generate descriptions for every image, adding cost and dependence on a proprietary model.

Experimental setup

Datasets: Twitter2015 Twitter2017

Benchmarks: Twitter2015 Twitter2017

Metrics: Micro-F1 Precision Recall Accuracy Macro-F1

Baselines: RAN UMT OCSGA JML VLP-MABSA CMMT AoM DQPSA GPT-4o ChatGPT-3.5 LLaMA2-13B Mixtral-AWQ

Builds on: TomBERT DeBERTa CLIP MFB bilinear pooling VLP-MABSA

Closest papers by meaning

• Aspect Enhancement and Text Simplification in Multimodal Aspect-Based Sentiment Analysis for Multi-Aspect and Multi-Sentiment Scenarios (AAAI 2025 · similarity 0.90)
• A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis (AAAI 2024 · similarity 0.89)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
• DIEM: Decomposition-Integration Enhancing Multimodal Insights (CVPR 2024 · similarity 0.86)
• SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.86)
• Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (AAAI 2025 · similarity 0.86)