Research map / Image generation / Autoregressive and latent generation
Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Psychometry trains one unified model on fMRI data from all NSD subjects, using a mixture-of-experts layer with subject-specific parameters and a retrieval-based inference step (Ecphory) to reconstruct images with Versatile Diffusion.
Problem
Image reconstruction from fMRI currently requires separate subject-specific models, and naively pooling data across subjects degrades performance, so inter-subject commonality is not exploited.
Approach
An Omni MoE layer where all experts process every subject's data (dense, unrouted) with per-subject learnable parameters and a split-then-lump token compression for efficiency, plus Ecphory, which mixes predicted CLIP embeddings with the nearest prestored subject-specific training embeddings before conditioning a pretrained diffusion model.
Key results
- Psychometry trained once on pooled subjects reaches 96.8% CLIP and 0.297 PixCorr, versus MindEye's unified-model 85.0% CLIP and 0.129 PixCorr.
- Psychometry improves over subject-specific MindEye by 2.0%/2.7%/2.6%/6.0% on the four high-level metrics (per the paper's text).
- Ablation: removing Ecphory drops CLIP score from 96.8% to 94.7%, and removing subject-specific parameters drops it to 87.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Psychometry instead of MindEye's unified model | NSD reconstruction · NSD test | CLIP: 85.0% -> 96.8% | Helped |
| use Psychometry instead of MindEye's unified model | NSD reconstruction · NSD test | PixCorr: .129 -> .297 | Helped |
| add Omni MoE with subject-specific parameters and Ecphory to the ViT baseline (ablation) | Psychometry · NSD test | CLIP: 81.6% -> 96.8% | Helped |
| add subject-specific parameters to Omni MoE (ablation) | Psychometry · NSD test | CLIP: 87.2% -> 96.8% | Helped |
| add Ecphory at inference (ablation) | Psychometry · NSD test | CLIP: 94.8% -> 96.8% | Helped |
Surprising findings
- State-of-the-art MindEye degrades substantially when trained on pooled data from all subjects (e.g., PixCorr 0.309 to 0.129), contrary to expected data-scaling benefits.
Limitations
- Ecphory and Omni MoE add inference-time retrieval over stored memories and per-subject parameters, with data privacy concerns when pooling subjects' fMRI data.
- Evaluated only on four NSD subjects using the standard split.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MS-COCO
Benchmarks: Natural Scenes Dataset test split
Metrics: PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP EffNet-B SwAV
Baselines: Mind-Reader Mind-Vis Takagi et al. Gu et al. MindEye sparse MoE (top-K) dense MoE
Builds on: Mixture-of-Experts CLIP Versatile Diffusion Vision Transformer MindEye
Closest papers by meaning
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.88)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.87)
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.86)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.86)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.85)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.