Research map / Vision-language and multimodal / Multimodal LLMs
FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FaceBench, a hierarchical multi-view, multi-level facial attribute VQA benchmark (49,919 eval and 23,841 training pairs), evaluates 9-11 MLLMs and fine-tunes LLaVA-1.5-13B into Face-LLaVA.
Problem
Existing MLLM evaluations and face datasets lack comprehensive, fine-grained coverage of facial attributes across multiple views and hierarchical levels, so MLLM face perception is poorly measured.
Approach
A hierarchical facial attribute structure (5 views, 3 attribute levels, ~211 attributes, ~701 values) is used to generate four VQA question types from images with manual multi-annotator labels, plus a fine-tuned LLaVA-1.5-13B baseline (Face-LLaVA) trained on the training split.
Key results
- GPT-4o achieves the best overall score of 63.21% and Gemini-1.5-Pro 62.72%, while human performance reaches 67.38% overall.
- Face-LLaVA-13B reaches 61.16% overall, outperforming all open-source models evaluated (best open-source baseline MiniCPM-V-8B at 60.67%).
- GPT-4o lags human evaluators by about 7.98% on Appearance, Accessories and Surrounding views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · FaceBench | overall score: 53.66 -> 61.16 | Helped |
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · Appearance | score: 51.22 -> 60.29 | Helped |
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · Accessories | score: 58.45 -> 61.62 | Helped |
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · Surrounding | score: 44.11 -> 57.89 | Helped |
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · Psychology | score: 58.47 -> 61.85 | Helped |
| Fine-tune LLaVA1.5-13B on 23,841 FaceBench VQA pairs | LLaVA1.5-13B · Identity | score: 68.10 -> 71.64 | Helped |
Surprising findings
- Face-LLaVA underperforms on some attributes such as earrings and background despite fine-tuning, attributed to baseline limitations and limited images.
- Commercial GPT-4o scores only 2.40% on background and 70.26% on gender, showing inconsistent per-attribute performance.
Limitations
- Face-LLaVA performance is limited by baseline and small number of training images, lagging on some attributes.
- Low resolution of RAF-DB, RAF-AU and FairFace images prevented human evaluation on Psychology and Identity views.
- OEQ answers synthesized by ChatGPT and response matching relies on ChatGPT judging, introducing potential evaluation noise.
Experimental setup
Datasets: FairFace RAF-DB RAF-AU CelebA-HQ FFHQ
Benchmarks: FaceBench
Metrics: accuracy macro F1 ROUGE-L
Baselines: Qwen-VL-Chat InstructBLIP DeepSeek-VL Qwen2-VL Molmo InternVL2 MiniCPM-V LLaMA-3.2-11B-Vision LLaVA-1.5-13B GPT-4o Gemini-1.5-Pro
Builds on: LLaVA FairFace RAF-DB RAF-AU CelebA-HQ
Closest papers by meaning
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.88)
- GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model (CVPR 2025 · similarity 0.88)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.87)
- MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions (AAAI 2026 · similarity 0.87)
- Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.