Research map / Image generation / Text-to-image diffusion
MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MindCustomer blends visual brain signals (fMRI from NSD) with image and optional text contexts in a mask-free, single-image generation pipeline built on Versatile Diffusion, using an Image-Brain Translator for cross-subject brain data augmentation.
Problem
Multi-context image generation that fuses brain-signal (fMRI) context with image and text contexts, handling cross-subject brain encoding under scarce data, cross-modal fusion, and preservation of each context's semantics.
Approach
An Image-Brain Translator (IBT) generates pseudo-fMRI from images to augment cross-subject brain embedding training; the diffusion model is fine-tuned on the transferred image context, then frozen while the brain embedding is optimized toward the image context, and the two embeddings are combined by linear interpolation (or concatenation for dual contexts).
Key results
- Quantitative comparison: MindCustomer achieves CLIP-I 0.563, DINOv2 0.654, CLIP-IQA 0.892, versus Baseline-1 at 0.559/0.630/0.734 and Baseline-2 at 0.526/0.612/0.830.
- User study (22 participants, 220 answers): quality score 2.618 vs 1.868 for Baseline-1, consistency 1.868 vs 1.675.
- Few-shot on Subj1 with 10%/20%/40% data: average score 0.703/0.713/0.722; IBT pseudo-fMRI averages Pearson 0.426 with ground truth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MindCustomer's IBT-based cross-modal fusion pipeline instead of direct visual-stimulus blending | Versatile Diffusion multi-context generation · Multi-context image generation | CLIP-I: 0.559 -> 0.563 | Helped |
| Use MindCustomer's IBT-based cross-modal fusion pipeline instead of reconstructed-image blending | Versatile Diffusion multi-context generation · Multi-context image generation | CLIP-I: 0.526 -> 0.563 | Helped |
| Add IBT, diffusion fine-tuning, and brain embedding optimization (ablation) | MindCustomer · Multi-context image generation | CLIP-I: 0.518 -> 0.563 | Helped |
| Add IBT (ablation) | MindCustomer · Multi-context image generation | CLIP-I: 0.515 -> 0.563 | Helped |
| Add diffusion fine-tuning and brain embedding optimization (ablation) | MindCustomer · Multi-context image generation | CLIP-I: 0.529 -> 0.563 | Helped |
| Add brain embedding optimization (ablation) | MindCustomer · Multi-context image generation | CLIP-I: 0.548 -> 0.563 | Helped |
| Add ClipCap semantic enhancement (ablation) | MindCustomer · Multi-context image generation | CLIP-I: 0.560 -> 0.563 | Helped |
| Increase few-shot Subj1 training data from 10% to 40% | MindCustomer · Subj1 few-shot generation | CLIP-I: 0.523 -> 0.561 | Helped |
| Increase few-shot Subj1 training data from 10% to 40% | MindCustomer · Subj1 few-shot generation | DINOv2: 0.642 -> 0.652 | Helped |
| Increase few-shot Subj1 training data from 10% to 40% | MindCustomer · Subj1 few-shot generation | CLIP-IQA: 0.944 -> 0.952 | Helped |
| Add IBT to brain representation pre-training (ablation) | MindCustomer brain decoding · Brain decoding across four NSD subjects | SSIM: .272 -> .289 | Helped |
Surprising findings
- Brain-decoding reconstruction of MindCustomer (PixCorr 0.157) is below UMBRAE (0.283), showing the brain-decoding component is not SOTA; the paper frames brain decoding as secondary to the blending task.
Limitations
- Brain signals are coarse and noisy (low SNR), limiting detailed or specific control compared with text and image contexts.
- Relies on limited NSD data from few subjects; brain data scarcity constrains precision and diversity.
- Brain decoding quality lags SOTA on low-level metrics such as PixCorr.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MS-COCO
Benchmarks: NSD test set (982 images)
Metrics: CLIP-I DINOv2 CLIP-IQA PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP Pearson correlation
Baselines: Baseline-1 (versatile diffusion with stimulus images) Baseline-2 (MindEye fMRI reconstruction + versatile diffusion) MindBridge UMBRAE
Builds on: Versatile Diffusion MindBridge ClipCap SoftCLIP DreamBooth
Closest papers by meaning
- MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning (AAAI 2025 · similarity 0.86)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.85)
- CognitionCapturer: Decoding Visual Stimuli from Human EEG Signal with Multimodal Information (AAAI 2025 · similarity 0.85)
- Mind Artist: Creating Artistic Snapshots with Human Thought (CVPR 2024 · similarity 0.84)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.84)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.