Research map / Video / Video-language understanding

Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception

NeurIPS 2026 · Video › Video-language understanding · approach: Video-language pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces behavioral geometric supervision (BGS), a LoRA-based hybrid triplet + RSA objective that aligns video foundation models to 49,484 human odd-one-out social video similarity judgments, closing the gap to caption-based MPNet and near the noise ceiling.

Problem

Video foundation models fail to match human similarity judgments for dynamic social interaction videos, underperforming caption-based sentence embeddings; it is unclear whether human social similarity structure can be induced in video models via scalable behavioral supervision and whether it is dist

Approach

A differentiable hybrid objective that combines a local triplet loss on human odd-one-out judgments with a global differentiable RSA loss matching pairwise cosine-distance geometry to the human RSM, applied via LoRA to several ViT backbones, with a matched MPNet-distillation control to isolate the supervision source.

Key results

• V-JEPA 2 ViT-L fine-tuned with BGS reaches R^2 = 0.172 vs 0.134 for the best sentence-embedding baseline (MPNet) and a split-half ceiling of 0.213; V-JEPA 2.1 ViT-B reaches 0.165 (78% of ceiling) up from 0.058 pretrained.
• Language-distillation control with MPNet targets reaches R^2 = 0.123, below the hybrid BGS fine-tuned models.
• On zero-shot PHASE transfer, BGS-aligned V-JEPA 2.1 ViT-B reaches 81% accuracy vs 55% pretrained baseline; UCF101 accuracy for V-JEPA 2.1 rises from 78.17% to 80.31%.

Surprising findings

• None of 14+ pretrained vision encoders matched caption-based MPNet embeddings on human social video similarity, despite the task being purely visual.
• The matched language-distillation control (MPNet targets) failed to reproduce the social-affective and transfer gains of behavioral supervision.

Limitations

• Single source corpus (Moments in Time) and predominantly Western, U.S. university participant pool limit cultural generality.
• Aggregate-consensus similarity data smooths over individual differences; small 250-video stimulus set.
• Test RSM covers only 637 of 1,225 pairs and the split-half noise ceiling is modest, so absolute gains are small in some cases (e.g., video-unique variance 0.02 to 0.04).

Experimental setup

Datasets: Moments in Time (curated social subset, 250 clips) PHASE UCF101 social-affective annotations (Moments in Time subset)

Benchmarks: social video odd-one-out similarity (RSA R^2) PHASE zero-shot 3-way social interaction classification UCF101 linear probe social-affective ridge probes

Metrics: Spearman R^2 (RSA) odd-one-out accuracy Top-1 accuracy Pearson r variance partitioning R^2 bootstrap CI

Baselines: paraphrase-multilingual-mpnet-base-v2 X3D-M CLIP ViT-B/32 Qwen3-VL-2B-Instruct SocialGNN VisualRNN SiMPLE

Builds on: RSA (representational similarity analysis) triplet loss LoRA VICE DreamSim

Closest papers by meaning

• Spot The Ball: A Benchmark for Visual Social Inference (CVPR 2026 · similarity 0.87)
• Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.86)
• Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.86)
• PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.86)
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.86)
• SocialGesture: Delving into Multi-person Gesture Understanding (CVPR 2025 · similarity 0.86)