Research map / Vision-language and multimodal / Multimodal LLMs
Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Micro-Bench, a expert-curated VQA/captioning benchmark of 17,235 microscopy images (perception plus cognition) to evaluate VLMs, and finds specialist models underperform, fine-tuning causes forgetting, and weight merging helps.
Problem
Existing biomedical vision-language benchmarks are narrow, mostly histopathology or radiology, and lack coverage of perception and cognition across microscopy modalities, scales, and organisms.
Approach
Builds a diverse microscopy VLM benchmark from permissively licensed public datasets with expert-curated metadata, closed VQA for coarse and fine-grained perception, bounding-box/segmentation localization, and an expert-submitted cognition set converted to multiple choice with GPT-4. It then uses the benchmark to show catastrophic forgetting in specialist fine-tuning and that weight merging with b
Key results
- GPT-4o reaches 62.68% on coarse-grained perception, 51.73% on fine-grained perception, and 62.00% on cognition, showing high error rates.
- BiomedCLIP is the best biomedical contrastive model at 34.65% fine-grained perception, below general CLIP-family and autoregressive models on coarse-grained tasks.
- Weight merging of PLIP/QuiltNet with base OpenCLIP/CLIP improves Micro-Bench performance over fine-tuned counterparts without additional training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o rather than the best biomedical contrastive model, BiomedCLIP | Micro-Bench coarse-grained perception · Micro-Bench coarse-grained perception | accuracy | Helped |
| Use GPT-4o rather than the best biomedical contrastive model, BiomedCLIP | Micro-Bench fine-grained perception · Micro-Bench fine-grained perception | accuracy | Helped |
| Use GPT-4o rather than the best biomedical contrastive model, BiomedCLIP | Micro-Bench cognition · Micro-Bench cognition (reasoning) | accuracy | Helped |
| Use a general-domain model rather than a biomedical specialist | Micro-Bench coarse-grained perception · Micro-Bench coarse-grained perception | accuracy | Mixed |
| Fine-tune OpenCLIP on pathology data to obtain PLIP (ablation) | OpenCLIP · Micro-Bench coarse-grained perception | accuracy: 36.34 -> 31.11 | Hurt |
| Fine-tune CLIP on pathology data to obtain QuiltNet (ablation) | CLIP · Micro-Bench coarse-grained perception | accuracy: 35.41 -> 26.58 | Hurt |
| Fine-tune OpenCLIP on pathology data to obtain PLIP (ablation) | OpenCLIP · Micro-Bench cognition | accuracy: 38.33 -> 17.00 | Hurt |
| Fine-tune CLIP on pathology data to obtain QuiltNet (ablation) | CLIP · Micro-Bench cognition | accuracy: 28.00 -> 13.00 | Hurt |
| Fine-tune pathology models on pathology data (ablation) | OpenCLIP and CLIP · Pathology-specific fine-grained perception | accuracy | Helped |
| Merge fine-tuned pathology-model weights with their pretrained base-model weights (ablation) | PLIP and QuiltNet · Micro-Bench perception | accuracy | Helped |
| Use autoregressive models rather than contrastive models | Micro-Bench evaluation · Coarse-grained perception | accuracy | Helped |
| Use contrastive models rather than autoregressive models | Micro-Bench evaluation · Fine-grained perception | accuracy | Helped |
Surprising findings
- Specialist biomedical fine-tuning often underperforms general models, with catastrophic forgetting on non-target tasks.
- Training-free weight merging with base models recovers general performance and improves specialist results.
Limitations
- Cognition subset is small (54 images, 121 questions) from 6 users.
- Closed VQA format with GPT-4-generated distractors may not reflect open-ended use.
- Coverage gaps in basic microscopy research acknowledged by authors.
Experimental setup
Datasets: Micro-Bench perception (26 public datasets) Micro-Bench cognition (54 images, 121 questions)
Benchmarks: Micro-Bench
Metrics: accuracy bootstrap confidence interval GRIT localization metric
Baselines: GPT-4o CogVLM QwenVLM PaliGemma ALIGN OpenCLIP CLIP BiomedCLIP CONCH PLIP QuiltNet
Builds on: CLIP BiomedCLIP PLIP CONCH
Closest papers by meaning
- MorphoHELM: A Comprehensive Benchmark for Evaluating Representations for Microscopy-Based Morphology Assays (NeurIPS 2026 · similarity 0.89)
- MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · similarity 0.88)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.88)
- BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · similarity 0.88)
- ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy (ICML 2025 · similarity 0.87)
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.