Research map / Vision-language and multimodal / Human and embodied modeling
JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JRDB-Social, a three-level (individual, intra-group, social group) annotation extension of the JRDB robot dataset, and benchmarks six video/image multimodal LLMs on it, finding they struggle at group-level context.
Problem
Existing datasets capture human behaviour at only one level (individual attributes, pairwise interactions, or group/video captions), and it is unknown how well multimodal LLMs understand layered social group behaviour in crowded robot-view scenes.
Approach
Annotates the JRDB robot dataset at three levels: individual demographics (gender, age, race), frame-level multi-label pairwise interactions within groups, and group-level text descriptions covering body position, salient scene content, venue, and purpose. Evaluates multimodal LLMs by converting outputs to classification via guided (cropped) and holistic (counting/binary) prompting.
Key results
- In guided perception, MiniGPT-4 averaged 0.2816 F1 and Video-LLaMA 13B 0.3025 overall, with performance dropping at group level.
- Counting-based holistic evaluation dropped sharply; best average F1 was 0.2135 (InstructBLIP 7B) in Table 2 terms of overall set, with Video-LLaMA 13B at 0.1965 in the second table variant.
- Binary holistic prompting gave higher F1 (up to 0.4848 average for InstructBLIP 7B) yet models still struggled at intra-group and social group levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the binary Yes/No task instead of holistic counting | Multi-modal LLMs on JRDB-Social · Overall | F1 score | Helped |
| Use frame-level crops at scale 2.5 instead of fixed black-mask or fixed without-mask crops (ablation) | MiniGPT-4 (LLaMA-2 7B) on social group analysis · Social Group Level | F1 score average: 0.1452 | Helped |
| Increase Video-LLaMA from LLaMA-2 7B to LLaMA-2 13B | Video-LLaMA on JRDB-Social · Overall (Guided Perception) | F1 score: 0.2874 -> 0.3025 | Mixed |
| Increase Video-LLaMA from LLaMA-2 7B to LLaMA-2 13B | Video-LLaMA on JRDB-Social · Overall (Holistic Counting) | F1 score: 0.1645 -> 0.1965 | Helped |
| Increase Video-LLaMA from LLaMA-2 7B to LLaMA-2 13B | Video-LLaMA on JRDB-Social · Overall (Holistic Binary) | F1 score: 0.4564 -> 0.4622 | Hurt |
Surprising findings
- Binary yes/no holistic queries outperformed counting queries, which the authors attribute to hallucination in the text encoder.
- Valley and OTTER were excluded from the guided table due to poor performance.
Limitations
- Evaluated only off-the-shelf multimodal LLMs with prompting; no fine-tuning on the dataset.
- Race annotation is subjective and based on a historical classification scheme.
- Dataset is captured on a university campus, limiting diversity of scenes.
Experimental setup
Datasets: JRDB
Benchmarks: JRDB-Social
Metrics: F1 score accuracy
Baselines: Video-LLaMA VALLEY OTTER MiniGPT-4 InstructBLIP
Builds on: JRDB LLaMA-2 MiniGPT-4 InstructBLIP Video-LLaMA
Closest papers by meaning
- JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments (CVPR 2024 · similarity 0.85)
- Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots (ICLR 2024 · similarity 0.84)
- JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics (AAAI 2026 · similarity 0.84)
- SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction (CVPR 2024 · similarity 0.84)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.84)
- COOPERA: Continual Open-Ended Human-Robot Assistance (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.