Research map / Detection and segmentation / Open-set and few-shot detection
Be More Specific: Evaluating Object-centric Realism in Synthetic Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines object-centric realism (OcR) with shape and texture scores, releases a human-annotated dataset of synthetic object images from four T2I models, and proposes OLIP, a frozen-BLIP-based regression model that beats existing realism/quality/alignment metrics.
Problem
Existing synthetic image evaluation metrics measure holistic realism or text alignment and are not granular, so they cannot assess realism of individual objects in generated images, nor align well with human perception.
Approach
Ground realism evaluation on individual objects, scoring each object's shape and texture realism separately on a 1-5 Likert scale, and train a dedicated regression architecture (OLIP) with a frozen BLIP backbone, full-image-query attention heads for shape/texture/joint, and MLP projectors.
Key results
- OLIP achieves 0.118 MAE and 0.539 accuracy on joint OcR, outperforming the best baseline VQAScore (0.164 MAE, 0.407 accuracy).
- Average object realism score correlates 0.79 (Spearman) with holistic image realism; the five largest objects predict holistic realism well (R2 = 0.62).
- Ablations show removing the full-image query, multi-head design, or deep projection each drops accuracy by 6-10%, and removing all drops it to 0.387.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add OLIP's full-image attention query (ablation) | OLIP · Object realism test set — Joint | Accuracy: 0.479 -> 0.539 | Helped |
| use separate prediction heads for shape, texture, and joint realism (ablation) | OLIP · Object realism test set — Joint | Accuracy: 0.486 -> 0.539 | Helped |
| use a deeper projection head instead of a linear projection (ablation) | OLIP · Object realism test set — Joint | Accuracy: 0.446 -> 0.539 | Helped |
| add full-image attention, separate prediction heads, and a deeper projection head (ablation) | OLIP · Object realism test set — Joint | Accuracy: 0.387 -> 0.539 | Helped |
| use OLIP instead of the strongest baseline | Object realism estimation · Object realism test set — Joint | Accuracy: 0.414 -> 0.539 | Helped |
| use a joint shape-and-texture prompt rather than averaging separate shape and texture prompts | Prompt-based methods · Object realism test set — Joint | Accuracy: 0.407 -> 0.414 | Helped |
| use few-shot rather than zero-shot evaluation | LLaVA · Object realism test set — Joint | Accuracy: 0.375 -> 0.414 | Helped |
Surprising findings
- FLUX-dev ranks higher than real images in realism scores.
- Existing realism metrics such as ImageReward and QualiCLIP perform close to random on object realism.
Limitations
- Limited number of image types and object categories (17 categories), so generalization to real-world scale is untested.
- Per-object evaluation requires multiple model passes and object grounding, adding computational cost.
- Low inter-annotator agreement (Cohen's Kappa 0.4 shape, 0.34 texture) reflects subjectivity of realism judgments.
Experimental setup
Datasets: Object-centric realism dataset (own, built from online product images and synthetic images)
Benchmarks: Object-centric realism dataset (own, 5% test split)
Metrics: MAE accuracy (5-class) Spearman correlation
Baselines: ImageReward AestheticScore QualiCLIP VQAScore LLaVA-OV (zero-shot) LLaVA-Critic (few-shot) Random
Builds on: BLIP VQAScore ImageReward LLaVA-Critic
Closest papers by meaning
- Position: What makes an image realistic? (ICML 2024 · similarity 0.85)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.85)
- 3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism (NeurIPS 2026 · similarity 0.84)
- BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model (NeurIPS 2024 · similarity 0.84)
- CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI (CVPR 2025 · similarity 0.83)
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.