Research map / Vision-language and multimodal / Multimodal LLMs
GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FacePlayGround-240K, a pixel-grounded face caption/QA dataset, and GroundingFace, a GLaMM-based model with HQ-SAM adapter, face-prior token sampler, two-stage LoRA and LoRA MoE for fine-grained face part grounding.
Problem
Current MLLMs with pixel grounding perform poorly on fine-grained face understanding, lacking face-specific grounded captions, QA data, and fine-grained face part masks.
Approach
Builds a large face dataset with 47 fine-grained part mask categories aligned to captions and hierarchical QA, and adapts GLaMM with an HQ-SAM shallow-deep adapter, a face-prior token sampler reusing SAM features, two-stage LoRA training, and a LoRA MoE routing high- and low-quality tokens.
Key results
- GroundingFace reaches 23.1 METEOR and 88.9 gIoU on face caption and referring segmentation versus 1.1 and 6.6 for GLaMM.
- On zero-shot face attribute recognition, GroundingFace (7B) scores 91.7 on RAF-DB versus 67.2 for InternVL-v1.5 (26B).
- On grounded caption and VQA, GroundingFace improves over the Stage-1 model (21.9 vs 19.8 METEOR on grounded caption).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune GLaMM on FacePlayGround-240K | GLaMM · Face caption | METEOR: 1.1 -> 18.2 | Helped |
| fine-tune GLaMM on FacePlayGround-240K | GLaMM · Face referring segmentation | gIoU: 6.6 -> 83.6 | Helped |
| add HQ-SAM Adapter, HQ facial tokens, and MoE (ablation) | Stage-2 · Face caption | METEOR: 21.3 -> 23.1 | Helped |
| add HQ-SAM Adapter, HQ facial tokens, and MoE (ablation) | Stage-2 · Face referring segmentation | gIoU: 88.2 -> 88.9 | Helped |
| use HQ-SAM Adapter and HQ facial tokens (ablation) | Stage-1 · Face caption | METEOR: 18.2 -> 20.6 | Helped |
| use HQ-SAM Adapter and HQ facial tokens (ablation) | Stage-1 · Face referring segmentation | gIoU: 83.6 -> 86.2 | Helped |
| add HQ facial tokens (ablation) | Stage-2 with HQ-SAM Adapter and MoE · Face caption | METEOR: 17.5 -> 23.1 | Helped |
| add HQ facial tokens (ablation) | Stage-2 with HQ-SAM Adapter and MoE · Face referring segmentation | gIoU: 87.1 -> 88.9 | Helped |
| add HQ-SAM Adapter (ablation) | Stage-2 with HQ facial tokens and MoE · Face caption | METEOR: 22.3 -> 23.1 | Helped |
| add HQ-SAM Adapter (ablation) | Stage-2 with HQ facial tokens and MoE · Face referring segmentation | gIoU: 85.3 -> 88.9 | Helped |
| use 6th, 12th, 18th, and 24th SAM features instead of 6th and 24th (ablation) | GroundingFace · Face caption | METEOR: 23.1 -> 22.3 | Hurt |
| use 6th, 12th, 18th, and 24th SAM features instead of 6th and 24th (ablation) | GroundingFace · Face referring segmentation | gIoU: 88.9 -> 88.1 | Hurt |
Surprising findings
- Using additional SAM features from 12th and 18th layers degraded performance relative to the default 6th and 24th layer features.
Limitations
- Baseline comparisons are limited since no prior pixel-grounding face works exist.
- Some makeup categories (highlighter, blush) were omitted due to fuzzy boundaries.
Experimental setup
Datasets: FacePlayGround-240K CelebAMask-HQ FFHQ EasyPortrait LAION-Face
Benchmarks: FacePlayGround-240K test split RAF-DB LFWA AgeDB
Metrics: METEOR AP50 mIoU gIoU accuracy
Baselines: GLaMM Qwen-VL InstructBLIP LLaVA-v1.5 InternVL-v1.5
Builds on: GLaMM HQ-SAM LISA MoE-LoRA
Closest papers by meaning
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.89)
- GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · similarity 0.88)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.88)
- FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs (CVPR 2025 · similarity 0.88)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.