Research map / Vision-language and multimodal / Human and embodied modeling
Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-body
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Fine-grained Human-body Abnormality Detection (FHAD) and HumanCalibrator, which fine-tunes LLaVA-1.5 7B to detect absent body parts and uses an inpainting model plus grounding to find and repair redundant or absent parts in generated human images and videos.
Problem
Text-to-image and text-to-video models often generate human bodies with absent or redundant body parts; existing VLMs perform near chance at detecting these abnormalities, and prior repair methods are coarse-grained and alter the whole image.
Approach
Train an absent-body detector by automatically masking body parts in real COCO images so the VLM learns part-correlation cues, and detect redundant parts by checking whether a diffusion inpainting model's regenerated semantics differ from the original part; then iteratively inpaint detected abnormalities while leaving other content unchanged.
Key results
- On AIGC Human-Aware 1K, HumanCalibrator reaches 80.69% average absent-part accuracy versus 19.37% for the best open VLM baseline (InternVL2-26B) and 25.82% for CLIP-Large-14 (Table 1).
- On redundant-part detection, HumanCalibrator averages 58.57% accuracy versus 22.75% for CLIP-Large-14 and 16.56% for LLaVA-34B.
- Repair keeps visual consistency with FID 16.55 versus 98.86 for the pose-conditioned baseline, and latent consistency 0.964 versus 0.668.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HumanCalibrator instead of CLIP-Large-14 for detecting absent abnormalities | AIGC Human-Aware 1K · AIGC Human-Aware 1K, absent abnormalities | accuracy: 25.82% -> 80.69% | Helped |
| Use HumanCalibrator instead of CLIP-Large-14 for detecting redundant abnormalities | AIGC Human-Aware 1K · AIGC Human-Aware 1K, redundant abnormalities | accuracy: 22.75% -> 58.57% | Helped |
| Use HumanCalibrator rather than the original images | HumanCalibrator repair · Repair quality | Human Concept Score: 22.59 -> 22.77 | Helped |
| Use HumanCalibrator rather than the original images | HumanCalibrator repair · Repair quality | CLIP Score: 41.87 -> 41.97 | Helped |
| Use HumanCalibrator rather than the original images | HumanCalibrator repair · Repair quality | Human CLIP Score: 26.36 -> 26.42 | Helped |
| Use HumanCalibrator instead of pose-conditioned repair | Visual consistency evaluation · Visual consistency | FID: 98.86 -> 16.55 | Helped |
| Use HumanCalibrator instead of pose-conditioned repair | Visual consistency evaluation · Visual consistency | Latent Consistency: 0.668 -> 0.964 | Helped |
Surprising findings
- Current powerful VLMs (GPT-4o, InternVL2, LLaVA-34B) perform close to random guessing at detecting absent or redundant body parts in human photos, even though humans find the task easy.
- Using larger VLMs and more training data did not substantially improve abnormality perception; accuracy was near chance across InternVL2 2B to 26B and GPT-4o.
Limitations
- Predefined set of 12 abnormality classes limits generalizability to other categories and abnormality types.
- The circular detect-and-inpaint framework is complex and limits stability.
- Repair quality metric improvements are modest, and redundant-part localization relies on inpainting behavior and thresholds.
Experimental setup
Datasets: COCO VidProM
Benchmarks: COCO Human-Aware Val AIGC Human-Aware 1K
Metrics: accuracy false discovery rate CLIP Score Human CLIP Score Human Concept Score FID Latent Consistency
Baselines: LLaVA-34B InternVL2-26B GPT-4o CLIP-Large-14 HumanRefiner
Builds on: LLaVA-1.5 HumanRefiner
Closest papers by meaning
- Human Body Restoration with One-Step Diffusion Model and A New Benchmark (ICML 2025 · similarity 0.84)
- CosmicMan: A Text-to-Image Foundation Model for Humans (CVPR 2024 · similarity 0.83)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.83)
- MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts (NeurIPS 2024 · similarity 0.82)
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.82)
- MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.