Research map / Vision-language and multimodal / Human and embodied modeling
MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindTuner, a cross-subject fMRI visual decoding framework that combines LoRA and non-linear Skip-LoRA adapters for new-subject fine-tuning plus an image-pivot fMRI-to-text retrieval step that corrects reconstruction semantics on NSD.
Problem
Cross-subject visual decoding from fMRI with very little new-subject training data (about 1 hour), where linear alignment ignores subject-specific visual fingerprints and text-based alignment yields poor semantic correction.
Approach
Injects LoRA and non-linear Skip-LoRA adapters into a multi-subject MindEye2-style fMRI encoder to learn each subject's visual fingerprint, and uses images as a pivot modality so fMRI is matched to category text prompts (80 COCO nouns) to semantically correct SDXL reconstructions.
Key results
- With 1 hour of data, MindTuner reaches 83.1% image retrieval and 76.0% brain retrieval, versus 79.0% and 57.4% for MindEye2, and outperforms MindEye2 on all reconstruction metrics.
- With 40 hours of data, MindTuner achieves the best high-level metrics (Incep 95.6%, CLIP 93.8%, Eff 0.612, SwAV 0.340) and is on par with MindEye2 on retrieval (98.9% image, 98.3% brain).
- Adding LoRAs and Skip-LoRAs (+12.3M trainable parameters, about 0.56% of the 2.2B model) improves brain retrieval from 77.6% to 87.4% on subject 1 with 1 hour of data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MindTuner instead of MindEye2 | Cross-subject visual decoding · NSD, 1 hour | Brain retrieval: 57.4% -> 76.0% | Helped |
| Use MindTuner instead of MindEye2 | Cross-subject visual decoding · NSD, 40 hours | CLIP: 93.0% -> 93.8% | Helped |
| Add LoRAs to the fine-tuned model (ablation) | MindEye2 · NSD, Subject 1, 1-hour fine-tuning | PixCorr: 0.235 -> 0.261 | Helped |
| Add Skip-LoRAs alongside LoRAs (ablation) | MindEye2 with LoRAs · NSD, Subject 1, 1-hour fine-tuning | PixCorr: 0.261 -> 0.264 | Helped |
| Add the adaptive projector (ablation) | MindEye2 · NSD, Subject 1, 1-hour fine-tuning | CLIP: 80.8% -> 81.2% | Helped |
| Add the adaptive projector (ablation) | MindEye2 · NSD, Subject 1, 1-hour fine-tuning | PixCorr: 0.235 -> 0.233 | Hurt |
| Add an activation function to the ridge regression head (ablation) | MindEye2 · NSD, Subject 1, 1-hour fine-tuning | PixCorr: 0.235 -> 0.191 | Hurt |
| Set LoRA and Skip-LoRA rank to 4 instead of 8 (ablation) | MindTuner · NSD, Subject 1, 1-hour fine-tuning | CLIP: 84.6% -> 84.6% | No clear effect |
| Set LoRA and Skip-LoRA rank to 16 instead of 8 (ablation) | MindTuner · NSD, Subject 1, 1-hour fine-tuning | CLIP: 84.6% -> 85.2% | No clear effect |
| Remove semantic correction (ablation) | MindTuner · NSD, 40 hours | PixCorr: 0.280 -> 0.322 | Hurt |
| Remove semantic correction (ablation) | MindTuner · NSD, 1 hour | PixCorr: 0.208 -> 0.224 | Hurt |
| Fine-tune the adaptive projector on COCO image-text pairs | OpenCLIP projector · COCO 2017 validation | Image-to-text retrieval: 81.3% -> 95.2% | Helped |
Surprising findings
- Adding an activation function to the ridge head to make alignment non-linear caused severe overfitting, with all metrics dropping sharply.
- Semantic correction improves semantic clarity but lowers high-level metrics; images directly from SDXL unCLIP scored higher at the high level than the corrected images.
Limitations
- Semantic correction uses only a single category noun from 80 COCO classes, so complex scenes (especially with people) are misdescribed.
- Color distribution of reconstructions is hard to control and was not addressed.
- Subject 7 underperforms and the neuroscience interpretation of the visual fingerprint remains partly unexplained.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) COCO 2017
Benchmarks: Natural Scenes Dataset (NSD)
Metrics: PixCorr SSIM AlexNet two-way identification Inception CLIP two-way identification EfficientNet-B1 distance SwAV distance image retrieval accuracy brain retrieval accuracy
Baselines: MindEye2 MindEye1 Takagi et al. Ozcelik et al. UMBRAE MindBridge
Builds on: MindEye2 LoRA MixCo SDXL unCLIP GIT
Closest papers by meaning
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.92)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.89)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.89)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.89)
- Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding (AAAI 2025 · similarity 0.89)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.