Research map / Image generation / Editing and style transfer
MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindPainter, a cross-modal self-supervised pipeline that paints masked natural images conditioned on fMRI visual-stimulus signals in a single diffusion model, using a Pseudo Brain Generator and Brain Adapter.
Problem
Brain-conditioned image editing on natural images is underexplored; the two-stage approach (fMRI-to-image decoding followed by image-conditioned editing) is slow, biased in brain interpretation, and yields unnatural edits.
Approach
Train a diffusion model to reconstruct masked source images from pseudo-brain signals generated from the complementary image by an MLP (Pseudo Brain Generator), with a Brain Adapter aligning brain embeddings to CLIP image space and a multi-mask policy (inpaint, outpaint, random) for generalization; inference uses real fMRI directly in one pass.
Key results
- In a 20-participant user study, MindPainter scored 2.31 on consistency and 2.36 on alignment versus 1.98 and 2.11 for the MindEye + Paint by Example baseline; quality was essentially tied (2.47 vs 2.49).
- Ablations show the Diffusion Prior alone scores 1.27 quality and w/o PBG drops quality to 1.49, while adding in & out masking raises quality to 2.29.
- Pseudo-fMRI predictions closely match ground-truth voxel values and CLIP-based semantic clustering of real and pseudo-fMRI embeddings is well-separated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MindPainter instead of the baseline | Brain-conditioned image painting · User study | Quality: 2.49 ± 0.62 -> 2.47 ± 0.62 | No clear effect |
| Use MindPainter instead of the baseline | Brain-conditioned image painting · User study | Consistency: 1.98 ± 0.82 -> 2.31 ± 0.75 | Helped |
| Use MindPainter instead of the baseline | Brain-conditioned image painting · User study | Alignment: 2.11 ± 0.81 -> 2.36 ± 0.76 | Helped |
| Add a diffusion prior rather than using the full MindPainter pipeline (ablation) | MindPainter · User study | Quality: 2.47 ± 0.62 -> 1.27 ± 0.53 | Hurt |
| Add a diffusion prior rather than using the full MindPainter pipeline (ablation) | MindPainter · User study | Consistency: 2.31 ± 0.75 -> 1.37 ± 0.66 | Hurt |
| Add a diffusion prior rather than using the full MindPainter pipeline (ablation) | MindPainter · User study | Alignment: 2.36 ± 0.76 -> 1.15 ± 0.47 | Hurt |
| Use complementary-image pseudo-brain signals rather than simulated fMRI of the entire source image (ablation) | MindPainter · User study | Quality: 1.49 ± 0.55 -> 2.47 ± 0.62 | Helped |
| Use complementary-image pseudo-brain signals rather than simulated fMRI of the entire source image (ablation) | MindPainter · User study | Consistency: 1.32 ± 0.56 -> 2.31 ± 0.75 | Helped |
| Use complementary-image pseudo-brain signals rather than simulated fMRI of the entire source image (ablation) | MindPainter · User study | Alignment: 1.39 ± 0.54 -> 2.36 ± 0.76 | Helped |
| Add outpaint masking to inpaint masking (ablation) | MindPainter · User study | Quality: 2.17 ± 0.80 -> 2.29 ± 0.71 | Helped |
| Add outpaint masking to inpaint masking (ablation) | MindPainter · User study | Consistency: 2.08 ± 0.82 -> 2.17 ± 0.67 | Helped |
| Add outpaint masking to inpaint masking (ablation) | MindPainter · User study | Alignment: 1.54 ± 0.54 -> 2.02 ± 0.73 | Helped |
Surprising findings
- The baseline matched MindPainter on image quality in the user study (2.49 vs 2.47).
Limitations
- Evaluation relies on a small user study (20 participants, 100 pairs) and qualitative examples without standard metrics.
- Subject-specific models are trained per NSD subject; only subject 1 results are presented.
- Efficiency gains are asserted but not quantified.
Experimental setup
Datasets: NSD OpenImages MS-COCO
Benchmarks: NSD test set fMRI with OpenImages source images (user study, 100 pairs)
Metrics: user-rated quality consistency alignment (1-3 scale)
Baselines: MindEye + Paint by Example (two-stage baseline) Diffusion Prior MindPainter w/o PBG
Builds on: Paint by Example MindEye Stable Diffusion CLIP
Closest papers by meaning
- Mind Artist: Creating Artistic Snapshots with Human Thought (CVPR 2024 · similarity 0.90)
- MindCustomer: Multi-Context Image Generation Blended with Brain Signal (ICML 2025 · similarity 0.86)
- Neural-Driven Image Editing (NeurIPS 2025 · similarity 0.85)
- Learning from Pattern Completion: Self-supervised Controllable Generation (NeurIPS 2024 · similarity 0.84)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.83)
- PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.