Research map / Vision-language and multimodal / Multimodal LLMs

From Perception to Punchline: Empowering VLM with the Art of In-the-wild Memes

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HUMOR, which trains VLMs for meme punchline generation via hierarchical multi-path chain-of-thought supervision, a group-wise pairwise reward model (EBC ranking), and GRPO-based RL.

Problem

Meme punchline generation is open-ended and subjective; direct image-to-caption supervision yields fluent but shallow, unfunny captions, and humor preferences are noisy and not comparable across templates.

Approach

Hierarchical multi-path CoT (template-level intent then context-level paths, with one path anchored to ground truth), a pairwise reward model trained only within groups sharing a template, aggregated by Expected Borda Count, and GRPO-style group-wise RL, supported by propositions bounding expected humor.

Key results

• HUMOR-CoT on Qwen2.5-7B reaches 91.5% human rate versus 75.7% for the base model and 91.3% for GPT-4o.
• HUMOR-CoT achieves humor score 2.68 vs 2.39 for Qwen2.5-7B-Instruct and readability 3.70, the highest among compared systems.
• Reward model (HUMOR-RM) on Keye-VL-8B reaches Kendall tau 0.78 with p<0.001 across templates, versus near-chance base model results.

Surprising findings

• Qwen2.5-VL-7B reward models prefer captions mentioning salient objects, whereas Keye-VL-8B prefers captions reflecting implied internal states, and base-model choice strongly affects rank consistency.
• HUMOR-CoT on 7B outperforms the 32B Qwen2.5 variant on the human-likeness and intent-preservation evaluations.

Limitations

• Results on the Qwen2.5-VL backbones for the reward model are mixed and weaker than Keye-VL-8B.
• VLM-based absolute scoring is unreliable, so the main evaluation relies on human annotation and a single judge model (Gemini-2.5-pro).
• Theoretical propositions rely on strong assumptions (bounded humor, mass of star paths) that are not empirically verified.

Experimental setup

Datasets: public meme dataset (over 4,000 images) MemeCap New Yorker caption contest benchmark MCC

Benchmarks: in-house human-evaluated meme test set 20 unseen meme templates

Metrics: human evaluation (humor, readability, relevance, originality text-level similarity context-swap distance human rate (%) Kendall's tau Spearman correlation

Baselines: Qwen2.5-7B-Instruct Qwen2.5-32B-Instruct InternVL3-8B GLM-4.1V-9B-Thinking Keye-VL-8B-preview GPT-4o Gemini-2.5-flash CoT with Single Path CoT with Self-Improve CoT with Subquestion

Builds on: GRPO chain-of-thought prompting

Closest papers by meaning

• HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation (CVPR 2026 · similarity 0.92)
• Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning (NeurIPS 2024 · similarity 0.90)
• Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions (NeurIPS 2024 · similarity 0.86)
• Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps (ICLR 2025 · similarity 0.86)
• MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.85)
• Talk Funny! A Large-Scale Humor Response Dataset with Chain-of-Humor Interpretation (AAAI 2024 · similarity 0.85)