Research map / Vision-language and multimodal / Human and embodied modeling
CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMEL, which captures metaphorical alignment in memes for multimodal emotion recognition via schematic-prompt conditional generation and curriculum-annealed disentangled contrastive matching.
Problem
Multimodal emotion recognition over metaphorical content such as memes, where emotions arise from non-literal cross-modal associations and context-dependent meaning that metaphor-agnostic methods miss.
Approach
Formulates schematic prompts from two metaphor theories (CMT and SPV) with generative MLE supervision for metaphorical analogy, plus a Gumbel-softmax disentangled contrastive matching loss whose temperature is annealed by a self-paced curriculum to keep predictions context-sensitive.
Key results
- CAMEL-S reaches 86.28 W-F1 on metaphor identification, 2.85 points above the second-best BLIP-C.
- On emotion-related SA and OD tasks, CAMEL-S gets 31.47 and 70.34 W-F1, versus 27.31 and 67.91 for BLIP-C.
- Human evaluation on 111 memes shows CAMEL-S has the highest validity proportion (63.96%) versus Flamingo (50.45%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAMEL-S instead of BLIP-C | MER · MI | M-F1: 83.43 -> 86.28 | Helped |
| Use CAMEL-S instead of BLIP-V | MER · MTC | M-F1: 59.18 -> 62.44 | Helped |
| Use CAMEL-C instead of BLIP-C | MER · SA | M-F1: 27.31 -> 31.54 | Helped |
| Use CAMEL-S instead of BLIP-C | MER · OD | M-F1: 67.91 -> 70.34 | Helped |
| Use CAMEL-S instead of BLIP-C | MER · ID | M-F1: 39.19 -> 42.31 | Helped |
| Add contextual semantic adaption (ablation) | CAMEL · SA | W-F1: 27.82 -> 31.54 | Helped |
| Add contextual semantic adaption (ablation) | CAMEL · OD | W-F1: 68.17 -> 70.34 | Helped |
| Add contextual semantic adaption (ablation) | CAMEL · ID | W-F1: 39.72 -> 42.31 | Helped |
| Add disentangled contrastive matching (ablation) | CAMEL · SA | W-F1: 28.92 -> 31.54 | Helped |
| Add curricular annealing (ablation) | CAMEL · SA | W-F1: 29.72 -> 31.54 | Helped |
| Add disentangled matching loss LDe (ablation) | CAMEL · SA | W-F1: 30.43 -> 31.54 | Helped |
| Add generative loss LG (ablation) | CAMEL · SA | W-F1: 28.22 -> 31.54 | Helped |
Surprising findings
- Ablating the schematic-prompt metaphor knowledge module reduces W-F1 more than ablating the disentangled matching module, showing metaphor knowledge matters most.
- The SPV-based variant (CAMEL-S) lags slightly in grammar fluency but wins on validity, suggesting soft alignment generalizes better than hard CMT alignment.
Limitations
- Assumes each meme expresses only one kind of contextual semantics; compound expressions are not handled.
- Human evaluation is limited to 111 valid memes after excluding refusals from some API models.
Experimental setup
Datasets: MET-Meme MemeCap
Benchmarks: MET-Meme MemeCap
Metrics: Accuracy weighted F1 human rating of grammar, validity, effectiveness
Baselines: ResNet BERT RoBERTa BERTweet EFcapt CLIP TOT BLIP Flamingo
Builds on: BLIP Transformer Gumbel-Softmax Conceptual Metaphor Theory Preference Semantics
Closest papers by meaning
- Uncertain Multimodal Intention and Emotion Understanding in the Wild (CVPR 2025 · similarity 0.87)
- Multi-Granular Multimodal Clue Fusion for Meme Understanding (AAAI 2025 · similarity 0.86)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.86)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.86)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.86)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.