Research map / Vision-language and multimodal / Human and embodied modeling
Human-Aligned Image Models Improve Visual Decoding from the Brain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces standard image encoders with human-aligned versions (DreamSim, gLocal, Harmonization) as frozen targets for contrastive EEG/MEG/fMRI-to-image retrieval, with DreamSim giving up to 21% top-1 gains.
Problem
Visual decoding from brain signals (EEG, MEG, fMRI) by retrieving viewed images, where prior work has focused on brain encoders and auxiliary losses but not on the image encoder used as the alignment target.
Approach
Use frozen image encoders fine-tuned on human perceptual similarity judgments (especially DreamSim, trained on NIGHTS) as the embedding space for InfoNCE alignment with brain signals, hypothesizing that mid-level perceptual attributes match rapid RSVP brain responses.
Key results
- Human-aligned (DreamSim) image encoders improve 200-way top-1 EEG retrieval with NICE by up to 21% over unaligned encoders, with significance reported for most backbones (p<0.001 for most).
- Across EEG architectures, human alignment raises average top-1 to about 62% versus 46% for unaligned encoders (aggregate across architectures and backbones as reported).
- On cross-subject NICE EEG decoding, the average top-1 improvement is small (e.g., CLIP 11.4% vs 7.9%), and on MEG the aligned average top-1 is 21.9% vs 18.6% for the Ensemble encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | NICE EEG visual retrieval · Things EEG2 200-way retrieval | top-1 accuracy: 46.15 -> 62.53 | Helped |
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | EEGNetV4 image retrieval · Things EEG2 200-way retrieval | top-1 accuracy: 33.94 -> 39.51 | Helped |
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | EEGConformer image retrieval · Things EEG2 200-way retrieval | top-1 accuracy: 18.05 -> 26.25 | Helped |
| use human-aligned Ensemble embeddings instead of unaligned Ensemble embeddings | NICE EEG cross-subject retrieval · Things EEG2 cross-subject 200-way retrieval (Avg.) | top-1 accuracy: 12.9±1.04 -> 12.8±1.37 | Mixed |
| use human-aligned CLIP embeddings instead of unaligned CLIP embeddings | NICE EEG cross-subject retrieval · Things EEG2 cross-subject 200-way retrieval (Avg.) | top-1 accuracy: 7.9±0.75 -> 11.4±0.87 | Helped |
| use human-aligned OpenCLIP embeddings instead of unaligned OpenCLIP embeddings | NICE EEG cross-subject retrieval · Things EEG2 cross-subject 200-way retrieval (Avg.) | top-1 accuracy: 8.6±0.58 -> 10.8±0.74 | Helped |
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | NICE MEG visual retrieval · Things MEG2 200-way retrieval (Avg.) | top-1 accuracy: 16.8±2.75 -> 23.2±5.18 | Helped |
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | MLP fMRI visual retrieval · NSD 1000-way retrieval (Avg.) | top-1 accuracy: 48.2±4.3 -> 58.2±4.6 | Helped |
| replace unaligned image embeddings with Dreamsim human-aligned embeddings | MLP fMRI visual retrieval · NSD 1000-way retrieval (Avg.) | top-5 accuracy: 78.7±3.1 -> 86.7±3.6 | Helped |
| use Dreamsim alignment rather than gLocal or Harmonization | NICE EEG visual retrieval · Things EEG2 retrieval | top-1 accuracy | Helped |
| use gLocal human alignment instead of unaligned embeddings | image retrieval from EEG · Things EEG2 retrieval | top-1 accuracy | No clear effect |
| use Harmonization human alignment instead of unaligned embeddings | image retrieval from EEG · Things EEG2 retrieval | top-1 accuracy | No clear effect |
Surprising findings
- gLocal and Harmonization alignment give little or no benefit over unaligned encoders, suggesting the alignment dataset (NIGHTS mid-level similarity vs. THINGS odd-one-out or saliency tasks) matters more than alignment per se.
- Harmonization-aligned models underperform unaligned ones for LeViT and ConvNext.
Limitations
- Evaluation limited to retrieval over small image sets (200-way for EEG/MEG), not image reconstruction.
- Cross-subject transfer gains are not significant for most image encoders.
- Relies on a single RSVP dataset (THINGS-EEG2) for the main EEG results and few MEG participants without statistical tests.
Experimental setup
Datasets: THINGS-EEG2 THINGS-MEG NSD NIGHTS
Benchmarks: THINGS-EEG2 200-way retrieval THINGS-MEG 200-way retrieval NSD 1000-way fMRI retrieval
Metrics: top-1 accuracy top-5 accuracy
Baselines: NICE ATM-S EEGNetV4 EEGConformer gLocal Harmonization
Builds on: InfoNCE NICE DreamSim gLocal Harmonization
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.91)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.90)
- What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.89)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.88)
- Neural‑Visual Decoding via Cognitive‑guided Adaptive Blurring and Information‑Constrained Alignment (NeurIPS 2026 · similarity 0.88)
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.