Research map / Image generation / Text-to-image diffusion
Hummingbird: High Fidelity Image Generation via Multimodal Context Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Hummingbird fine-tunes SDXL's UNet via LoRA with rewards from a BLIP-2 QFormer-based Multimodal Context Evaluator so that generated images preserve scene attributes of a reference image and text guidance while remaining diverse.
Problem
Diffusion-based generators produce diverse images but lose fine-grained scene attributes (count, position, color, interactions) from a reference image given a text query, which hurts synthetic data for scene-aware tasks like VQA and HOI reasoning.
Approach
Use an MLLM to write a context description of the reference image focused on query-relevant attributes, then fine-tune SDXL's UNet with LoRA by backpropagating a global semantic reward (max cosine similarity between BLIP-2 visual query tokens and text CLS token) and a fine-grained ITM reward from the BLIP-2 QFormer, balancing fidelity with diversity.
Key results
- With LLaVA v1.6 7B on MME Perception, Hummingbird improves Position ACC+ by 13.34 points over the real-only baseline (66.67 vs 53.33).
- On Bongard-HOI with TPT, Hummingbird averages 68.73% accuracy, +2.14 over the best augmentation baseline (RandAugment 66.59 avg with I2T2I SDXL at 67.38).
- On ImageNet with TPT, Hummingbird averages 50.19 vs 42.26 real-only, and OOD average 47.09 vs 40.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Hummingbird image augmentation instead of I2T2I SDXL | LLaVA v1.6 7B on MME Perception Position · MME Perception, Position | ACC+: 50.00 -> 66.67 | Helped |
| use Hummingbird-generated images instead of I2T2I SDXL | CLIP-ResNet50 with TPT · Bongard-HOI average | Accuracy: +2.14% | Helped |
| use Hummingbird-generated images instead of Image Translation | ImageNet TPT · ImageNet | Top-1 accuracy: 61.90 -> 62.60 | Helped |
| add the Global Semantic Reward (ablation) | Hummingbird with both rewards · MME Perception, LLaVA v1.6 7B, Position | ACC+: 63.33 -> 66.67 | Helped |
| add the Fine-Grained Consistency Reward (ablation) | Hummingbird with both rewards · MME Perception, LLaVA v1.6 7B, Count | ACC+: 66.67 -> 70.00 | Helped |
| add fine-tuning (ablation) | Hummingbird · MME Perception, LLaVA v1.6 7B, Position | ACC+: 56.67 -> 66.67 | Helped |
| replace generic prompt with the designed prompt template (ablation) | Hummingbird with LLaVA v1.6 7B context descriptor · MME Perception, LLaVA v1.6 7B, Count | ACC+: 56.67 -> 70.00 | Helped |
| replace CLIP evaluator with BLIP-2 QFormer evaluator (ablation) | Hummingbird with LLaVA v1.6 7B context descriptor · MME Perception, LLaVA v1.6 7B, Count | ACC+: 66.67 -> 70.00 | Helped |
| use LLaVA instead of InternVL for context descriptions (ablation) | Hummingbird · MME Perception, LLaVA v1.6 7B evaluation, Count | ACC+: 70.00 vs 66.67 | Mixed |
| fine-tune Hummingbird (ablation) | Hummingbird image generator · Reference vs generated images | Euclidean distance: 36.10 -> 36.94 | No clear effect |
| use Hummingbird instead of I2T2I SDXL | Image generation for user-study MME images · MME Perception user study average | Fidelity selection percentage: 56.57 -> 76.06 | Helped |
Surprising findings
- Image Variation and Textual Inversion degrade VQA performance below the real-only baseline on MME, showing generic synthetic augmentation can hurt scene-aware tasks.
- RandAugment and Image Translation achieve lower FID than Hummingbird, since they sacrifice diversity, showing FID is not suited to this task.
Limitations
- Inherits limitations of SDXL, failing on symbolic or numerical reasoning and compositional multi-attribute prompts.
- MLLM context descriptors sometimes contain incorrect or ambiguous information, lowering fidelity.
- Evaluated on single attributes such as count, position, and color rather than multiple attributes at once.
Experimental setup
Datasets: VQAv2 GQA Bongard-HOI ImageNet MME Perception ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch
Benchmarks: MME Perception Bongard-HOI ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch MME Artwork MME Commonsense Reasoning
Metrics: ACC ACC+ Top-1 accuracy Euclidean distance (diversity) FID user study fidelity
Baselines: RandAugment Image Variation Image Translation (Boomerang) Textual Inversion I2T2I SDXL
Builds on: SDXL BLIP-2 LoRA DDIM Test-time prompt tuning
Closest papers by meaning
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.87)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.87)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.87)
- Generative Powers of Ten (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.