Research map / Large language models / Agents, code and math
ConnectomeBench: Can LLMs proofread the connectome?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConnectomeBench, a multimodal benchmark testing whether LLMs (Claude, o4-mini, GPT-4.1/4o, InternVL-3, NVLM) can perform segment type identification, split error correction, and merge error detection for connectome proofreading using MICrONS and FlyWire data.
Problem
Connectome proofreading (correcting segmentation split and merge errors) requires heavy expert human effort; it is unknown whether multimodal LLMs can perform the perceptual and judgment tasks involved.
Approach
Frame connectome proofreading subtasks as image-based benchmark questions built from proofreading edit histories (CAVEclient), evaluating frontier multimodal LLMs zero-shot with description and expert-derived heuristic prompts against human and fine-tuned ResNet-50 baselines.
Key results
- Segment identification balanced accuracy up to 0.822 on MICrONS (Claude Sonnet 3.7) and 0.529 on FlyWire (GPT-4.1), vs ~0.2-0.25 chance.
- Multiple-choice split error correction: o4-mini reaches 0.828 on FlyWire and 0.790 on MICrONS with description prompt; heuristics raise o4-mini MICrONS to 0.850.
- Merge error identification is hard: best binary accuracy is 0.628 (FlyWire) and 0.615 (MICrONS) for o4-mini, versus human 0.74-0.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add category descriptions to the prompt | proprietary LLMs for segment identification · FlyWire and MICrONS segment identification | balanced accuracy | No clear effect |
| Add description context to o4-mini | o4-mini binary split error correction · FlyWire | accuracy: 0.663 -> 0.631 | Hurt |
| Add description context to o4-mini | o4-mini binary split error correction · MICrONS | accuracy: 0.704 -> 0.679 | Hurt |
| Add description context to Claude Sonnet 4 | Claude Sonnet 4 multiple-choice split error correction · FlyWire | accuracy: 0.475 -> 0.657 | Helped |
| Add description context to GPT-4o | GPT-4o multiple-choice split error correction · MICrONS | accuracy: 0.620 -> 0.720 | Helped |
| Add split-error reasoning heuristics to the Description prompt | o4-mini binary split error correction · FlyWire | accuracy: 0.631 -> 0.754 | Helped |
| Add split-error reasoning heuristics to the Description prompt | o4-mini binary split error correction · MICrONS | accuracy: 0.679 -> 0.786 | Helped |
| Add split-error reasoning heuristics to the Description prompt | o4-mini multiple-choice split error correction · FlyWire | accuracy: 0.828 -> 0.788 | Hurt |
| Add split-error reasoning heuristics to the Description prompt | o4-mini multiple-choice split error correction · MICrONS | accuracy: 0.790 -> 0.850 | Helped |
| Add description context to o4-mini | o4-mini binary merge error identification · FlyWire | accuracy: 0.553 -> 0.628 | Helped |
| Add description context to o4-mini | o4-mini multiple-choice merge error identification · FlyWire | accuracy: 0.740 -> 0.670 | Hurt |
Surprising findings
- Adding category descriptions often did not improve proprietary models on segment identification, suggesting strong internal priors.
- The NEURD baseline detected none of the 14 merge error events it was evaluated on.
Limitations
- Merge error evaluation limited to a subset of 100 examples and split-error analysis to 100 examples; human baselines based on about 50 examples per condition.
- Open-weight models evaluated only on a subset of tasks; InternVL-3 78B run without dynamic tiling due to memory limits.
- Tasks do not cover synapse identification or merge error correction, which are needed for full proofreading.
Experimental setup
Datasets: MICrONS FlyWire
Benchmarks: ConnectomeBench
Metrics: balanced accuracy accuracy precision recall
Baselines: ResNet-50 (fine-tuned) Human expert NEURD Claude Sonnet 4 GPT-4o InternVL-3 NVLM
Builds on: CAVEclient RLCorrector NEURD
Closest papers by meaning
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.84)
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.83)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.83)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.83)
- MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation (NeurIPS 2026 · similarity 0.83)
- Do LLMs Signal When They’re Right? Evidence from Neuron Agreement (ICML 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.