Research map / Image generation / Editing and style transfer
Neural-Driven Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds L-Mind, a 23,928-pair dataset of multimodal neural signals (EEG, fNIRS, PPG, head motion) recorded during image-editing intent, and LoongX, a FLUX-based DiT editor conditioned on these signals via CS3 and DGF modules.
Problem
Image editing usually requires manual text prompts or physical inputs, which is inaccessible to users with motor or language impairments; it is unclear whether neural signals alone can drive instruction-based editing.
Approach
Encodes EEG, fNIRS, PPG and head-motion streams with a cross-scale state-space encoder (CS3), fuses them with a dynamic gated fusion module (DGF), pretrains encoders contrastively against text embeddings, and conditions a FLUX-based DiT via LoRA fine-tuning.
Key results
- Neural-signal-only LoongX reaches CLIP-I 0.6605 and DINO 0.4812, versus 0.6558 and 0.4636 for text-driven OminiControl.
- LoongX with neural signals plus speech reaches CLIP-T 0.2588, above text-only OminiControl at 0.2549.
- Ablation: EEG alone gives CLIP-I 0.5457; adding fNIRS raises DINO from 0.2963 to 0.4811.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace text conditioning with neural signals only | LoongX vs OminiControl (Text) · L-Mind test | CLIP-I: 0.6558 -> 0.6605 | Helped |
| Replace text conditioning with neural signals only | LoongX vs OminiControl (Text) · L-Mind test | DINO: 0.4636 -> 0.4812 | Helped |
| Replace text conditioning with neural signals only | LoongX vs OminiControl (Text) · L-Mind test | L1: 0.2632 -> 0.2509 | Helped |
| Add speech to neural signals | LoongX · L-Mind test | CLIP-T: 0.2436 -> 0.2588 | Helped |
| Add fNIRS to EEG (ablation) | LoongX modality combination · L-Mind test | DINO: 0.2963 to 0.4811 | Helped |
| Use the Oz EEG channel (ablation) | LoongX single-channel EEG · L-Mind test | CLIP-I: 0.6619 | Helped |
| Use the Oz EEG channel (ablation) | LoongX single-channel EEG · L-Mind test | DINO: 0.4873 | Helped |
| Use the Fpz EEG channel (ablation) | LoongX single-channel EEG · L-Mind test | CLIP-T: 0.2481 | Helped |
| Add CS3 without pretraining or DGF (ablation) | LoongX architecture · L-Mind test, neural signals+speech | CLIP-I: 0.5966 -> 0.6408 | Helped |
| Add CS3 without pretraining or DGF (ablation) | LoongX architecture · L-Mind test, neural signals+speech | CLIP-T: 0.2584 -> 0.2248 | Hurt |
| Add DGF without pretraining or CS3 (ablation) | LoongX architecture · L-Mind test, neural signals+speech | CLIP-T: 0.2584 -> 0.2620 | Helped |
| Add CS3 and DGF without pretraining (ablation) | LoongX architecture · L-Mind test, neural signals+speech | CLIP-I: 0.5966 -> 0.6319 | Helped |
Surprising findings
- Neural signals alone yield a slightly higher CLIP-I and DINO than text prompts, yet a lower CLIP-T (0.2436 vs 0.2549).
- The Oz channel gives the best global editing scores, and the frontal Fpz channel gives the best CLIP-T.
Limitations
- Only 12 healthy young participants; generalization to other populations is untested.
- Low-density EEG and fNIRS give limited spatial resolution.
- Results rely on a single text-based baseline (OminiControl) and small, within-CI margins on some metrics.
Experimental setup
Datasets: L-Mind SEED-Data-Edit
Benchmarks: L-Mind test set
Metrics: L1 L2 CLIP-I DINO CLIP-T multi-label mAP F1
Baselines: OminiControl (text) OminiControl (speech)
Builds on: diffusion transformer FLUX.1-dev OminiControl structured state space model
Closest papers by meaning
- MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning (AAAI 2025 · similarity 0.85)
- PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery (AAAI 2025 · similarity 0.84)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.84)
- Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.84)
- Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens (NeurIPS 2025 · similarity 0.84)
- CognitionCapturer: Decoding Visual Stimuli from Human EEG Signal with Multimodal Information (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.