Research map / Vision-language and multimodal / Multimodal LLMs
Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces YesBut, a 348-comic benchmark of two-panel contradictory comics with four tasks (literal description, contradiction generation, philosophy selection, title matching) and evaluates commercial and open-source VLMs and LLMs, finding a gap to human performance.
Problem
Large vision-language models struggle to understand humor arising from juxtaposition and nonlinear narratives in two-panel comics with contradictory content.
Approach
A benchmark of captionless "YES, BUT" comics annotated with literal descriptions, contradictions, underlying philosophies, and titles via a GPT-4-assisted human-in-the-loop pipeline, with tasks spanning literal comprehension to deep narrative reasoning.
Key results
- Claude-3 achieves best philosophy selection accuracy at 84.10%, while GPT-4 reaches 82.76%.
- LLaVA-1.6-34B achieves the best title matching accuracy at 63.31%, with GPT-4 at 60.25%.
- Open-source VLMs trail commercial models, and models lag behind human performance on the benchmark (human result not quantified in the text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a predicted literal description before title selection (ablation) | LLaVA-1.6-13B · YesBut title matching | accuracy: 55.08 -> 48.76 | Hurt |
| Add a predicted literal description before title selection (ablation) | Qwen-VL-Chat · YesBut title matching | accuracy: 42.05 -> 37.55 | Hurt |
| Add a predicted literal description before title selection (ablation) | mPlug-Owl2 · YesBut title matching | accuracy: 43.10 -> 37.84 | Hurt |
| Add a predicted literal description before title selection (ablation) | LLaVA-1.6-7B · YesBut title matching | accuracy: 37.07 -> 34.96 | Hurt |
| Add a predicted literal description before philosophy selection (ablation) | LLaVA-1.6-13B · YesBut philosophy selection | accuracy: 69.16 -> 68.68 | Hurt |
| Add a predicted literal description before philosophy selection (ablation) | Qwen-VL-Chat · YesBut philosophy selection | accuracy: 59.10 -> 59.58 | Helped |
| Add a predicted literal description before philosophy selection (ablation) | mPlug-Owl2 · YesBut philosophy selection | accuracy: 62.17 -> 60.25 | Hurt |
| Add a predicted literal description before philosophy selection (ablation) | LLaVA-1.6-7B · YesBut philosophy selection | accuracy: 47.41 -> 53.07 | Helped |
Surprising findings
- Decomposing the task by first generating a literal description and then reasoning often reduces title-matching accuracy for VLMs.
- LLMs given LLaVA-generated descriptions (no image) perform comparably to open-source VLMs on deep reasoning tasks.
Limitations
- Small dataset of 348 comics.
- Subjectivity of comic interpretation may introduce ambiguity in philosophy and title labels.
- Relies on GPT-4 for annotation drafts and GPT-based evaluation, which may introduce bias.
Experimental setup
Datasets: Anton Gudim YES, BUT comics scraped from social media
Benchmarks: YesBut
Metrics: accuracy BERTScore recall ROUGE-2 recall GPT-based evaluation score human rating (correctness, completeness, faithfulness)
Baselines: GPT-4 Claude-3 LLaVA-1.6-34B LLaVA-1.6-13B LLaVA-1.5-13B InstructBLIP-13B CogVLM Qwen-VL-Chat mPLUG-Owl2 LLaVA-1.6-7B InstructBLIP-7B ChatGPT
Builds on: GPT-4 LLaVA-1.6 BERTScore ROUGE
Closest papers by meaning
- HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation (CVPR 2026 · similarity 0.90)
- Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning (NeurIPS 2024 · similarity 0.88)
- Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps (ICLR 2025 · similarity 0.87)
- From Perception to Punchline: Empowering VLM with the Art of In-the-wild Memes (NeurIPS 2026 · similarity 0.86)
- SatireDecoder: Visual Cascaded Decoupling for Enhancing Satirical Image Comprehension (AAAI 2026 · similarity 0.86)
- Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.