Research map / Vision-language and multimodal / Cross-modal learning
A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DQPSA, which uses a prompt as both visual and language query (Prompt as Dual Query) and an Energy-based Pairwise Expert to jointly extract aspects and sentiments in multi-modal aspect-based sentiment analysis, reporting state-of-the-art on Twitter2015, Twitter2017 and Political Twitter.
Problem
Multi-modal aspect-based sentiment analysis (MATE, MASC, JMASA) suffers from differing visual focus across analysis targets, modal gap between separately encoded image and text, and span-based extractors ignoring pairwise relevance of span boundaries.
Approach
A prompt serves as both a visual query (via alternating self- and cross-attention over frozen CLIP image features, BLIP-2 style) and a language query for the text encoder, yielding prompt-aware visual information; an energy-based pairwise scoring of start-end boundaries replaces independent boundary prediction for span extraction.
Key results
- On JMASA, DQPSA reaches 71.9 F1 on Twitter2015 and 70.6 F1 on Twitter2017, beating the prior best AoM by 3.3 and 0.9 points.
- On MATE, DQPSA achieves 87.7 F1 (Twitter2015) and 94.3 F1 (Twitter2017), improving over the sub-optimal method by 1.5 and 2.0 points.
- On MASC, DQPSA reports 81.1 accuracy on Twitter2015 versus 75.9 for AoM... (reported F1 81.1 vs 75.9 F1 for AoM); it outperforms ChatGPT-3.5 (58.9 F1) and VisualGLM-6B (54.5 F1) on Twitter2017 MASC comparison table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2015 MATE | F1: 86.3 -> 87.7 | Helped |
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2015 MASC | F1: 80.9 -> 81.1 | Helped |
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2015 JMASA | F1: 69.1 -> 71.9 | Helped |
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2017 MATE | F1: 92.5 -> 94.3 | Helped |
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2017 MASC | F1: 73.1 -> 75.0 | Helped |
| Add EPE to DQPSA (ablation) | DQPSA · Twitter2017 JMASA | F1: 66.8 -> 70.6 | Helped |
| Use PDQ visual query instead of optimizable tokens (ablation) | DQPSA · Twitter2015 MASC | F1: 78.4 -> 81.1 | Helped |
| Use PDQ visual query instead of optimizable tokens (ablation) | DQPSA · Twitter2015 JMASA | F1: 69.9 -> 71.9 | Helped |
| Use PDQ visual query instead of optimizable tokens (ablation) | DQPSA · Twitter2017 MASC | F1: 69.7 -> 75.0 | Helped |
| Use PDQ visual query instead of optimizable tokens (ablation) | DQPSA · Twitter2017 JMASA | F1: 65.6 -> 70.6 | Helped |
| Add both PDQ and EPE (ablation) | DQPSA · Twitter2015 MATE | F1: 84.4 -> 87.7 | Helped |
| Add both PDQ and EPE (ablation) | DQPSA · Twitter2017 JMASA | F1: 64.1 -> 70.6 | Helped |
Surprising findings
- Replacing the prompt-based visual query with optimizable tokens barely hurts MATE (87.4 vs 87.7 F1 on Twitter2015) but substantially drops MASC and JMASA.
- The text-only PSA variant with EPE alone improves ASTE F1 on four datasets, e.g., 14lap from 65.6 to 69.8.
Limitations
- Twitter2017 MASC is weaker than AoM (75.0 vs 76.4 accuracy) and the authors attribute this to emojis and unresolvable symbols.
- Comparison to LLMs limited to MASC task only, and ChatGPT-3.5 receives text only.
- Code and pretrained models promised upon acceptance; no release link given in the text.
Experimental setup
Datasets: Twitter2015 Twitter2017 Political Twitter COCO2017 ImageNet ASTE-Data-V2
Benchmarks: Twitter2015 Twitter2017 Political Twitter ASTE-Data-V2
Metrics: Precision Recall Micro-F1 Accuracy
Baselines: SPAN D-GCN BART UMT+TomBERT OSCGA+TomBERT JML VLP-MABSA CMMT AoM RAN UMT OSCGA
Builds on: FSUIE BLIP-2 Energy-based Model BERT
Closest papers by meaning
- DEQA: Descriptions Enhanced Question-Answering Framework for Multimodal Aspect-Based Sentiment Analysis (AAAI 2025 · similarity 0.89)
- Aspect Enhancement and Text Simplification in Multimodal Aspect-Based Sentiment Analysis for Multi-Aspect and Multi-Sentiment Scenarios (AAAI 2025 · similarity 0.88)
- You Only Read Once: Constituency-Oriented Relational Graph Convolutional Network for Multi-Aspect Multi-Sentiment Classification (AAAI 2024 · similarity 0.86)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.85)
- SimRP: Syntactic and Semantic Similarity Retrieval Prompting Enhances Aspect Sentiment Quad Prediction (AAAI 2025 · similarity 0.85)
- Modeling Adaptive Inter-Task Feature Interactions via Sentiment-Aware Contrastive Learning for Joint Aspect-Sentiment Prediction (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.