Research map / Vision-language and multimodal / Multimodal LLMs

Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces S³, a seizure video dataset (438 videos, 116 patients, 20 ILAE semiological features) and a seven-task MLLM benchmark with a clinical report metric (Seizure-RQI), plus SFT/GRPO fine-tuning of Qwen2.5-Omni-7B and a two-stage feature-then-Random-Forest classifier.

Problem

MLLMs' ability to interpret involuntary, spatio-temporally evolving seizure semiology is untested, and no large expert-annotated seizure video dataset or clinically grounded evaluation exists.

Approach

A clinically annotated seizure video dataset with dense feature-level labels, a seven-task hierarchical benchmark (perception to diagnosis), a clinically weighted report metric (Seizure-RQI), and a two-stage pipeline that uses MLLM-extracted binary semiology features with a lightweight classifier for ES vs NES.

Key results

• Two-stage pipeline with seizure_omni_sft-7B reaches F1 0.96 on ES vs NES classification, versus 0.70 average F1 for direct MLLM classification.
• Seizure-RQI correlates with expert judgment at Pearson 0.57 and pairwise accuracy 0.74, while BLEU, ROUGE-L and BERTScore show Pearson r ≤ 0.10.
• SFT fine-tuning of Qwen2.5-Omni-7B raises Task 1 mean F1 from 0.38 to 0.47 and GRPO lowers average MAE from 25.50s to 20.02s.

Surprising findings

• Scaling Qwen models from 7B to 72B yields similar Task 1 mean F1 (~0.42–0.45), indicating size alone does not help.
• Seizure fine-tuning collapses Task 3 lateralization to a single direction, giving F1 = 0.00 for head turning after SFT, due to limited laterality data.

Limitations

• Single-center adult cohort from one EMU (UCLA).
• Low frame-rate processing (1–2 FPS) limits capture of rapid movements.
• Spatial laterality remains near-chance (mean F1 < 0.2) for general models and fine-tuning hurts Task 3.

Experimental setup

Datasets: S³ seizure video dataset (438 videos, 116 patients)

Benchmarks: S³ seven-task benchmark

Metrics: F1 precision recall accuracy MAE edit distance temporal F1 LCS ratio BLEU ROUGE-L

Baselines: InternVL3.5-8B Qwen2.5-VL-7B Qwen3-VL-8B InternVL3.5-38B Qwen2.5-VL-32B Qwen3-VL-32B Qwen2.5-VL-72B Qwen2.5-Omni-7B Qwen3-Omni-30B-A3B Lingshu-32B Audio-Flamingo-3

Builds on: GRPO Qwen2.5-Omni

Closest papers by meaning

• Long-Term EEG Partitioning for Seizure Onset Detection (AAAI 2025 · similarity 0.86)
• NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.86)
• DMNet: Self-comparison Driven Model for Subject-independent Seizure Detection (NeurIPS 2024 · similarity 0.85)
• MMVU: Measuring Expert-Level Multi-Discipline Video Understanding (CVPR 2025 · similarity 0.84)
• OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.83)
• REST: Efficient and Accelerated EEG Seizure Analysis through Residual State Updates (ICML 2024 · similarity 0.83)