Research map / Large language models / Language, safety and interpretability
L-Man: A Large Multi-modal Model Unifying Human-centric Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds HumanIns, a 2.5M-instruction language-image dataset from 20 human-centric datasets, and trains L-Man, a Vicuna-7B-based large multimodal model with a multi-grained query adapter and two-stage training for unified human-centric tasks.
Problem
General large multi-modal models lack human-related domain knowledge and fine-grained perception, while existing human-centric unified models are restricted to pre-defined output forms and lack open-ended instruction-following capability.
Approach
Converts human-centric annotations (detection, pose, attributes, expression, action, face) into open-ended instruction-answer pairs, and uses a query adapter that pulls features from multiple intermediate vision-encoder layers under instruction guidance, pretrained first with contrastive and masked LM objectives on image-text data before instruction tuning an LLM.
Key results
- On PA-100K, L-Man reaches 82.26 mA, 79.98 accuracy and 88.71 F1 with all components, versus 83.72 mA and 88.21 F1 for the specialist VTB (mA is higher for VTB).
- On RAF-DB expression recognition, L-Man averages 86.31% versus 86.31 vs 85.86 for Transfer, without task-specific fine-tuning.
- On UCF101, L-Man reaches 99.6% accuracy, matching VideoMAE V2 and exceeding other specialists; on Stanford40 it reaches 94.9%, above the listed specialists.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L-Man instead of the strongest listed LMM baseline | L-Man · PA100K | accuracy: 67.5 -> 84.2 | Helped |
| Add language-image contrastive learning (ablation) | L-Man · PA-100K | mA: 76.62 -> 78.76 | Helped |
| Add vision-grounded masked language modeling (ablation) | L-Man · PA-100K | mA: 76.62 -> 78.12 | Helped |
| Add both pretraining objectives (ablation) | L-Man · PA-100K | mA: 76.62 -> 79.82 | Helped |
| Add multi-grained visual feature extraction (ablation) | L-Man · PA-100K | mA: 76.62 -> 77.13 | Helped |
| Add language-image contrastive learning and multi-grained visual features (ablation) | L-Man · PA-100K | mA: 76.62 -> 79.64 | Helped |
| Add vision-grounded masked language modeling and multi-grained visual features (ablation) | L-Man · PA-100K | mA: 76.62 -> 78.56 | Helped |
| Add both pretraining objectives and multi-grained visual features (ablation) | L-Man · PA-100K | mA: 76.62 -> 81.34 | Helped |
| Add instruction-aware adaptation (ablation) | L-Man · PA-100K | mA: 76.62 -> 76.93 | Helped |
| Add language-image contrastive learning and instruction-aware adaptation (ablation) | L-Man · PA-100K | mA: 76.62 -> 78.96 | Helped |
| Add vision-grounded masked language modeling and instruction-aware adaptation (ablation) | L-Man · PA-100K | mA: 76.62 -> 78.55 | Helped |
| Add both pretraining objectives and instruction-aware adaptation (ablation) | L-Man · PA-100K | mA: 76.62 -> 80.24 | Helped |
Limitations
- LMM comparison relies on manual evaluation of 120 images per dataset, with no automatic metric.
- L-Man does not uniformly beat specialists; for example PA-100K mA and Stanford40 numbers trail some baselines.
- Only 128 query tokens and a single 7B LLM were explored.
Experimental setup
Datasets: HumanIns COCO CUHK-PEDES ICFG-PEDES UCF101 Stanford40 PA-100K RAF-DB AIC MPII CityPersons WiderFace
Benchmarks: Stanford40 UCF101 CityPersons COCO PA100K RAF-DB WiderFace AIC MPII
Metrics: accuracy precision recall F1 mA manual answer quality score
Baselines: InstructBLIP Qwen-VL-Chat LLaVA1.5 VTB VideoMAE V2 Transfer (RAF-DB)
Builds on: BLIP-2 LLaVA Vicuna CLIP
Closest papers by meaning
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.89)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.89)
- ModaVerse: Efficiently Transforming Modalities with LLMs (CVPR 2024 · similarity 0.89)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.