Research map / Image generation / Editing and style transfer
Mind Artist: Creating Artistic Snapshots with Human Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindArt, a two-stream fMRI decoder that uses graph-matching-guided optimal transport representation learning plus frozen GPT-2 and Stable Diffusion branches to reconstruct and style fMRI-evoked images without extra optimization.
Problem
Reconstructing realistic, semantically faithful images from non-invasive fMRI recordings is hard due to scarce image-fMRI annotations, and existing decoders treat fMRI data independently and lack controllable style.
Approach
Casts neural decoding as a discrete optimal transport / graph matching problem over dynamic multiplex graphs of fMRI and image entities (self-supervised, Sinkhorn-based), and pairs a GM-guided fMRI encoder with a frozen GPT-2 language stream and a latent diffusion visual stream so that style can be injected via text prompts training-free.
Key results
- On DIR, MindArt reaches CLIPT@50 of 26.0% and FID 2.67, vs 14.0% and 2.70 for MindGPT.
- On GOD, MindArt achieves CLIPScore 0.631 and CLIPT@50 24.0%, outperforming GESS and MinD-Vis on semantic metrics (MinD-Vis CLIPT@50 12.0%).
- Removing the graph-matching encoder drops CLIPT@10 from 43.6% to 32.0% on the MindArt-L variant (ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MindArt instead of MindGPT | DIR image reconstruction · DIR | CLIPT@50: 14.0% -> 26.0% | Helped |
| use MindArt instead of MinD-Vis | GOD image reconstruction · GOD | CLIPScore: 0.625±0.10 -> 0.631±0.13 | Helped |
| use MindArt instead of VQ-fMRI | GOD image reconstruction · GOD | SSIM: 0.423±0.11 -> 0.242±0.13 | Hurt |
| use MindArt instead of MindGPT | DIR language reconstruction · DIR | SPICE: 10.3 -> 11.3 | Helped |
| add graph matching-guided neural representation learning (ablation) | MindArt-L · GOD | CLIPT@50: 12.0% -> 24.0% | Helped |
| add graph matching-guided neural representation learning (ablation) | MindArt-L · GOD | SSIM: 0.198±0.13 -> 0.242±0.13 | Helped |
| use a 16-layer encoder instead of a 4-layer encoder (ablation) | MindArt · GOD | CLIPT@50: 18.0% -> 24.0% | Helped |
| use a 16-layer encoder instead of an 8-layer encoder (ablation) | MindArt · GOD | CLIPT@50: 20.0% -> 24.0% | Helped |
| use HVC voxels instead of VC voxels (ablation) | MindArt-L · GOD | CLIPT@50: 24.0% -> 26.0% | Helped |
| use LVC voxels instead of VC voxels (ablation) | MindArt-L · GOD | SSIM: 0.242±0.13 -> 0.239±0.14 | No clear effect |
| add predicted latent representation Xz (ablation) | MindArt-L · GOD | SSIM: 0.197±0.15 -> 0.242±0.13 | Helped |
| add predicted latent representation Xz (ablation) | MindArt-L · GOD | CLIPT@10: 43.7%±3.9% -> 43.6%±3.9% | No clear effect |
Surprising findings
- The lower-level latent input Xz contributes little to semantic metrics but gives a 22.8% SSIM gain, and HVC voxels give the best semantic scores while LVC gives better low-level similarity.
Limitations
- fMRI is noisy and costly to acquire, limiting data scale
- Generative-model reconstructions have limited low-level shape and posture control
- Evaluated only on two small public fMRI datasets with 8 subjects
Experimental setup
Datasets: DIR dataset GOD dataset ImageNet
Benchmarks: DIR GOD
Metrics: SSIM CLIPScore FID CLIPT@10 CLIPT@50 BLEU-1 BLEU-4 ROUGE METEOR SPICE
Baselines: Takagi et al. VQ-fMRI MindGPT MinD-Vis GESS
Builds on: Stable Diffusion MAGNN Sinkhorn entropic optimal transport MinD-Vis Takagi et al. latent diffusion decoding
Closest papers by meaning
- MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning (AAAI 2025 · similarity 0.90)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.87)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.87)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.86)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.85)
- Image Translation as Diffusion Visual Programmers (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.