Research map / Large language models / Language, safety and interpretability
SimRP: Syntactic and Semantic Similarity Retrieval Prompting Enhances Aspect Sentiment Quad Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Syn2Vec syntactic vectors over constituent subtrees plus SimRP, a two-stage retrieval (syntactic then semantic) of demonstrations for prompting generative and LLM models on aspect sentiment quad prediction, along with a CsT-based fuzzy metric.
Problem
Aspect Sentiment Quad Prediction requires recognizing aspect and opinion terms that depend on sentence syntax, but semantically similar demonstrations used in prompts do not supply syntactic cues; exact-match metrics also underestimate LLM performance.
Approach
Retrieve demonstrations first by syntactic similarity using IDF-weighted constituent-subtree vectors (Syn2Vec), then re-rank by Sentence-BERT semantic similarity, and build prompts from these demonstrations; additionally evaluate with a constituent-subtree-based fuzzy matching metric.
Key results
- In the SFT paradigm, SimRP with T5-large reaches 53.30 F1 on Rest15 and 63.42 F1 on Rest16 (average 58.36), improving the average by 2.20 points over the prior best IVLS (56.16).
- In the ICL paradigm, GPT-4o with SimRP reaches 43.17 F1 on Rest15 and 49.74 F1 on Rest16 at 10-shot under exact matching.
- Under the CsT-based fuzzy metric, GPT-4o with 10-shot SimRP reaches 59.25 on Rest15 and 63.12 on Rest16, comparable to fine-tuned generative models on Rest15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SimRP instead of the strongest reported baseline | T5-large ASQP model · Rest15 | F1: 51.28 -> 53.30 | Helped |
| use SimRP instead of the strongest reported baseline | T5-large ASQP model · Rest16 | F1: 61.04 -> 63.42 | Helped |
| use SimRP retrieval instead of LLMs for SA demonstration selection | GPT-4o (10-shot) · Rest15 | F1: 37.08 -> 43.17 | Helped |
| use SimRP retrieval instead of LLMs for SA demonstration selection | GPT-4o (10-shot) · Rest16 | F1: 45.00 -> 49.74 | Helped |
| add syntactic similarity retrieval (ablation) | SimRP with T5-large · Rest15 | F1: 52.58 -> 53.30 | Helped |
| add syntactic similarity retrieval (ablation) | SimRP with T5-large · Rest16 | F1: 62.97 -> 63.42 | Helped |
| add semantic similarity retrieval (ablation) | SimRP with T5-large · Rest15 | F1: 52.44 -> 53.30 | Helped |
| add semantic similarity retrieval (ablation) | SimRP with T5-large · Rest16 | F1: 62.83 -> 63.42 | Helped |
| replace random demonstrations with syntactically and semantically retrieved demonstrations (ablation) | SimRP with T5-large · Rest15 | F1: 51.00 -> 53.30 | Helped |
| replace random demonstrations with syntactically and semantically retrieved demonstrations (ablation) | SimRP with T5-large · Rest16 | F1: 62.18 -> 63.42 | Helped |
| add syntactic similarity retrieval (ablation) | GPT-4o + SimRP (1-shot) · Rest15 | F1: 36.48 -> 37.88 | Helped |
| add semantic similarity retrieval (ablation) | GPT-4o + SimRP (1-shot) · Rest16 | F1: 42.56 -> 45.50 | Helped |
Surprising findings
- Semantically similar demonstrations contribute more than syntactically similar ones in both SFT and ICL ablations.
- Exact-match F1 substantially underestimates LLM quadruple extraction; many errors are reasonable but more detailed or less detailed spans.
Limitations
- Fuzzy CsT-based metric relies on parsing quality and ground-truth extension; it is an evaluation heuristic rather than the official metric.
- More demonstrations hurt performance on Rest16 in SFT, suggesting overfitting to retrieved examples seen in training.
- Syn2Vec depends on spaCy constituent parsing.
Experimental setup
Datasets: Rest15 Rest16
Benchmarks: Rest15 Rest16
Metrics: precision recall F1 CsT-based fuzzy F1
Baselines: GAS Paraphrase DLO ILO MvP GenDA MUL OTCL IVLS ThoR LLMs for SA
Builds on: T5 Sentence-BERT Paraphrase generative ASQP MvP Retrieval-augmented generation
Closest papers by meaning
- Harnessing Holistic Discourse Features and Triadic Interaction for Sentiment Quadruple Extraction in Dialogues (AAAI 2024 · similarity 0.85)
- A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis (AAAI 2024 · similarity 0.85)
- Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations (AAAI 2024 · similarity 0.85)
- You Only Read Once: Constituency-Oriented Relational Graph Convolutional Network for Multi-Aspect Multi-Sentiment Classification (AAAI 2024 · similarity 0.84)
- Aspect Enhancement and Text Simplification in Multimodal Aspect-Based Sentiment Analysis for Multi-Aspect and Multi-Sentiment Scenarios (AAAI 2025 · similarity 0.84)
- Modeling Adaptive Inter-Task Feature Interactions via Sentiment-Aware Contrastive Learning for Joint Aspect-Sentiment Prediction (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.