Research map / Image generation / Autoregressive and latent generation
An Evaluation Framework for Product Images Background Inpainting Based on Human Feedback and Product Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HFPC, an automatic evaluator for AI background-inpainted product images that combines a BLIP-based image-referenced reward model trained on 44k human-labeled pairs with a fine-tuned EfficientSAM product-consistency module.
Problem
Evaluating AI background inpainting of product images is hard: existing metrics (FID, CMMD, CLIP-IQA, NIMA) do not match human judgments on inappropriate backgrounds or on product inconsistency between original and generated images, so filtering relies on costly manual annotation.
Approach
Combine a human-feedback reward model that scores the original-generated image pair (BLIP features, attention fusion, ranking plus classification loss) with a segmentation-based check that matches and compares product masks between original and generated images.
Key results
- HFPC reaches 0.964 precision (Pb) for low-quality filtering with the single reward model at 0.965, versus 0.905 for ImageReward and 0.890 for CLIP-Score.
- HFPC achieves the best PLCC (0.352) and SRCC (0.367) among compared methods, above ImageReward (0.271/0.302).
- Low-quality recall Rb improves to 0.403 for HFPC compared with 0.370 for ImageReward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add original-image information through attention-based fusion rather than using the ImageReward ranking model without original-image input (ablation) | BLIP reward model · HFPC-44k test | Pb: 0.905 -> 0.965 | Helped |
| Add original-image information through attention-based fusion rather than using the ImageReward ranking model without original-image input (ablation) | BLIP reward model · HFPC-44k test | Rb: 0.370 -> 0.387 | Helped |
| Add original-image information by feature concatenation (ablation) | BLIP reward model · HFPC-44k test | Pb: 0.905 -> 0.915 | Helped |
| Add original-image information by feature concatenation (ablation) | BLIP reward model · HFPC-44k test | Rb: 0.370 -> 0.355 | Hurt |
| Add classification loss (ablation) | BLIP reward model · HFPC-44k test | Pb: 0.953 -> 0.965 | Helped |
| Add classification loss (ablation) | BLIP reward model · HFPC-44k test | Rb: 0.342 -> 0.387 | Helped |
| Add product-consistency assessment to the reward model (ablation) | HFPC · HFPC-44k test | Rb: 0.387 -> 0.403 | Helped |
| Add product-consistency assessment to the reward model (ablation) | HFPC · HFPC-44k test | Pb: 0.965 -> 0.964 | Hurt |
Surprising findings
- The single reward model slightly outperforms the full HFPC on precision Pb (0.965 vs 0.964) and high-quality recall Rg (0.924 vs 0.922).
Limitations
- Product consistency module relies on a fine-tuned segmentation model trained on only about 500 manually annotated images.
- Product matching uses IoU and positional heuristics that assume similar shapes and arrangements across original and generated images.
- Evaluation is on a single proprietary dataset from business scenarios, and the inpainting generator used to produce the images is not specified.
Experimental setup
Datasets: HFPC-44k
Benchmarks: HFPC-44k
Metrics: PLCC SRCC Pb (low-quality filtering precision) Rb (low-quality recall) Rg (high-quality recall)
Baselines: CLIP-Score CLIP-IQA NIMA ImageReward Single Reward Model
Builds on: ImageReward BLIP EfficientSAM Grounding DINO CLIP-IQA
Closest papers by meaning
- HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images (CVPR 2026 · similarity 0.87)
- RefAdGen: High-Fidelity Advertising Image Generation (AAAI 2026 · similarity 0.84)
- PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference (NeurIPS 2024 · similarity 0.82)
- AI-generated Image Quality Assessment in Visual Communication (AAAI 2025 · similarity 0.82)
- ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding (CVPR 2026 · similarity 0.80)
- TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting (CVPR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.