Research map / Vision-language and multimodal / CLIP and zero-shot
Teaching Human Behavior Improves Content Understanding Abilities Of VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains LLaMA-Vid on 730k Reddit/YouTube images and videos to simulate receiver behavior (likes, comments, replay graphs), yielding Behavior-LLaVA, which improves zero-shot and fine-tuned content understanding across many VLM benchmarks.
Problem
Receiver behavior (likes, comments, replay graphs) is collected at scale but ignored when training vision-language models, even though it carries signals about content; it is unclear whether training on it improves content understanding.
Approach
Treat large-scale digital-analytics behavior (likes, views, top comments, replay graphs) as an instruction fine-tuning target for a VLM, so the model learns to simulate audience behavior for a given image/video, and then transfers those content-relevant signals to downstream content understanding tasks; an Ad-LLaVA control trained on the same media without behavior isolates the effect of behavior.
Key results
- Behavior-LLaVA improves over LLaMA-Vid on zero-shot emotion recognition, e.g., CAER 27.2 to 51.0 and Video Emotion-8 29.7 to 41.35, and on persuasion and action tasks in the video advertisement benchmarks.
- On the 7 memorability benchmarks, zero-shot Behavior-LLaVA beats LLaMA-Vid on all 7 (e.g., Memento10k 0.03 to 0.12), and with 25% fine-tuning data it nearly matches fine-tuned SOTA.
- On image dense captioning judged by GPT-4V, Behavior-LLaVA (13B) scores higher than LLaMA-Vid on Detail (8.1 vs 7.6) and Quality (7.9 vs 7.2), approaching LLaVA-1.6 (34B) on those aspects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BLIFT receiver behavior training | LLaMA-Vid · Video Emotion-8, CAER, and Ekman-6, 0-shot | accuracy: 29.7 -> 41.35; 27.2 -> 51.0; 37.33 -> 49.33 | Helped |
| Add BLIFT receiver behavior training | LLaMA-Vid · MSVD-QA, MSRVTT-QA, ActivityNet-QA | accuracy: 69.7 -> 70.1; 57.7 -> 59.2; 47.4 -> 47.5 | Mixed |
| Add BLIFT receiver behavior training | LLaMA-Vid · Image dense captioning | Correctness: 7.4 -> 7.3 | Hurt |
| Add BLIFT receiver behavior training | LLaMA-Vid · Image dense captioning | Detail: 7.6 -> 8.1 | Helped |
| Add BLIFT receiver behavior training | LLaMA-Vid · Image dense captioning | Quality: 7.2 -> 7.9 | Helped |
| Train on comments only (ablation) | LLaMA-Vid · CAER | accuracy: 75.6 -> 79.1 | Helped |
| Train on comments only (ablation) | LLaMA-Vid · MSRVTT-QA | accuracy: 58.9 -> 59.0 | No clear effect |
| Train on Salicon region saliency (ablation) | LLaMA-Vid · MSRVTT-QA | accuracy: 58.9 -> 55.6 | Hurt |
| Train on Salicon object saliency (ablation) | LLaMA-Vid · EmoSet | accuracy: 45.23 -> 48.0 | Helped |
| Use a 1:1 behavior-to-instruction sampling ratio for 2.2 epochs (ablation) | LLaMA-Vid with BLIFT · 6-task zero-shot evaluation | Performance: 0 -> 12.57 | Helped |
| Use a 1:10 behavior-to-instruction sampling ratio for 1.2 epochs instead of 1:1 for 2.2 epochs (ablation) | LLaMA-Vid with BLIFT · 6-task zero-shot evaluation | Performance: 12.57 -> 5.51 | Hurt |
| Add receiver behavior targets (comments, likes, and views) rather than content verbalization alone | LLaMA-Vid on BLIFT · Long Video Understanding, 9 tasks | average improvement over baseline: 21.49% | Helped |
Surprising findings
- Ad-LLaVA, trained on the same media without behavior, performs nearly identically to the base LLaMA-Vid, suggesting the gains come from the behavior signal rather than extra data.
- Perception-level behavior (Salicon saliency) gives little or negative gain on action recognition, while action-level behavior gives consistent improvements.
Limitations
- Tested only with LLaMA-Vid as base model, so generality across VLM architectures is untested.
- Behavior data comes from social platforms (Reddit, YouTube) and may carry biases; filtering is heuristic and manually tuned.
- Limitations section admits the hypothesis is tested only on comments and likes with limited benchmark breadth; GPT-4V judge used for captioning with no human evaluation.
Experimental setup
Datasets: BLIFT ShareGPT LLaVA SFT visual QA mix Salicon MSRVTT-QA MSVD-QA ActivityNet-QA LVU HVU VideoEmotion-8 Ekman-6 CAER
Benchmarks: LVU HVU MSVD-QA MSRVTT-QA ActivityNet-QA Video advertisement benchmarks (topic, sentiment, persuasion, action, reason) VideoEmotion-8 Ekman-6 CAER IAPSa Abstract Emotion6
Metrics: accuracy mAP R2 perplexity BLEU ROUGE METEOR correctness/detail/quality score (1-10)
Baselines: LLaMA-Vid Ad-LLaVA VideoChat Video4096 LCBM GPT4-V LLaVA-1.6 (34B) ViTMem Video-ChatGPT
Builds on: LLaMA-Vid LLaVA Video-ChatGPT
Closest papers by meaning
- Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior (ICLR 2024 · similarity 0.87)
- Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.84)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.84)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.84)
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.84)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.