Research map / Video / Video segmentation and tracking
EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the SEED-DV EEG-video dataset (20 subjects, 1400 clips, 40 concepts), an EEG visual perception benchmark, and EEG2Video, a Seq2Seq plus dynamic-aware noise-adding framework that reconstructs videos from EEG.
Problem
Decoding dynamic visual perception (video) from EEG, which has been studied mainly for static images and fMRI-based video reconstruction limited by low temporal resolution.
Approach
Use a Transformer Seq2Seq model over sliding-window EEG embeddings to predict VAE latents of video frames at high temporal resolution, plus a semantic predictor (CLIP text embeddings from BLIP captions) and a dynamic-aware noise-adding process that balances static and diverse noise based on decoded motion level, feeding an inflated (Tune-A-Video) Stable Diffusion model.
Key results
- On the EEG-VP benchmark, GLMNet reached 6.20% top-1 on 40-class concept classification (chance 2.50%) and 57.35% on color classification, with dynamics (fast/slow) decodable.
- EEG2Video reached 15.9% 40-way semantic accuracy and SSIM 0.256 on the full 40-class set; 34.0% 40-way accuracy and SSIM 0.300 on the 10-class subset.
- Ablations show removing the Seq2Seq module or DANA lowers 40-way video-level accuracy on the 40-class set (0.113 and 0.128 vs 0.159 for full model).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Seq2Seq (ablation) | EEG2Video · 40-class SEED-DV | Video semantic accuracy, 2-way: 0.786±0.03 -> 0.798±0.03 | Helped |
| add Seq2Seq (ablation) | EEG2Video · 40-class SEED-DV | Video semantic accuracy, 40-way: 0.113±0.01 -> 0.159±0.01 | Helped |
| add Seq2Seq (ablation) | EEG2Video · 40-class SEED-DV | Frame semantic accuracy, 2-way: 0.734±0.02 -> 0.774±0.02 | Helped |
| add Seq2Seq (ablation) | EEG2Video · 40-class SEED-DV | Frame semantic accuracy, 40-way: 0.112±0.01 -> 0.138±0.01 | Helped |
| add Seq2Seq (ablation) | EEG2Video · 40-class SEED-DV | SSIM: 0.189±0.03 -> 0.256±0.03 | Helped |
| add DANA (ablation) | EEG2Video · 40-class SEED-DV | Video semantic accuracy, 2-way: 0.770±0.02 -> 0.798±0.03 | Helped |
| add DANA (ablation) | EEG2Video · 40-class SEED-DV | Video semantic accuracy, 40-way: 0.128±0.01 -> 0.159±0.01 | Helped |
| add DANA (ablation) | EEG2Video · 40-class SEED-DV | Frame semantic accuracy, 2-way: 0.732±0.03 -> 0.774±0.02 | Helped |
| add DANA (ablation) | EEG2Video · 40-class SEED-DV | Frame semantic accuracy, 40-way: 0.109±0.03 -> 0.138±0.01 | Helped |
| add DANA (ablation) | EEG2Video · 40-class SEED-DV | SSIM: 0.217±0.02 -> 0.256±0.03 | Helped |
Surprising findings
- Number of main objects and human/face presence were not decodable from EEG above chance in the EEG-VP benchmark.
- Occipital electrodes were most informative for object recognition, while temporal-region electrodes carried motion (fast/slow) information.
Limitations
- Dataset recorded in a single session per subject, limiting study of stable neural patterns over time.
- Evaluation only in subject-dependent settings; cross-subject generalization unexplored.
- Low signal-to-noise ratio and spatial resolution make video decoding difficult; reconstruction performance is modest.
Experimental setup
Datasets: SEED-DV
Benchmarks: EEG-VP benchmark SEED-DV video reconstruction benchmark
Metrics: top-1 accuracy top-5 accuracy accuracy (2-way/40-way) SSIM chance-level t-test
Baselines: ShallowNet DeepNet EEGNet Conformer TSConv SVM MLP
Builds on: Tune-A-Video Stable Diffusion CLIP BLIP EEGNet
Closest papers by meaning
- What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.90)
- Neuro-3D: Towards 3D Visual Decoding from EEG Signals (CVPR 2025 · similarity 0.90)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.90)
- Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.90)
- Decoding Natural Images from EEG for Object Recognition (ICLR 2024 · similarity 0.89)
- Reanimating Images using Neural Representations of Dynamic Stimuli (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.