Research map / Vision-language and multimodal / Multimodal LLMs
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Quilt-Instruct, a 107K histopathology instruction-tuning dataset grounded by narrators' cursor traces in YouTube pathology videos, and trains Quilt-LLaVA with a new Quilt-VQA evaluation set.
Problem
Histopathology multimodal models lack spatially grounded instruction data and cross-patch reasoning over whole slide images, since existing data is patch-level image-caption pairs from PubMed.
Approach
Extract narrators' mouse cursor positions from educational histopathology videos, cluster them spatio-temporally with narration to create bounding-box-grounded captions, and generate instruction data with GPT-4 including reasoning prompts conditioned on WSI-level diagnosis and supporting facts.
Key results
- Quilt-LLaVA achieves 75.6 overall relative GPT-4 score on the instruction-following test set, versus 55.7 for LLaVA and 64.8 for LLaVA-Med.
- Quilt-LLaVA variants improve open-set VQA by about 4% and closed-set VQA by about 9% over the best baseline on average.
- Quilt-Instruct generation cost about $8,998 in GPT-4 API spend, covering 107K training pairs and 1,283 Quilt-VQA samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Quilt-LLaVA with Quilt-Instruct | Quilt-LLaVA · PathVQA and Quilt-VQA | Open recall: over 4% over SOTA | Helped |
| Train Quilt-LLaVA with Quilt-Instruct | Quilt-LLaVA · PathVQA and Quilt-VQA | Closed accuracy: average of 9% over SOTA | Helped |
| Increase Stage 1 training from 1 to 3 epochs (ablation) | Quilt-LLaVA with Quilt ViT-B-32 · Quilt-VQA | Open recall: 59.82 -> 60.97 | Helped |
| Increase Stage 2 training from 1 to 3 epochs (ablation) | Quilt-LLaVA with Quilt ViT-B-32 · Quilt-VQA | Open recall: 59.82 -> 57.62 | Hurt |
| Replace Quilt ViT-B-32 with ViT-B-16 vision encoder (ablation) | Quilt-LLaVA · Quilt-VQA | Closed accuracy: 64.43 -> 61.52 | Hurt |
| Replace Quilt ViT-B-32 with CLIP ViT-L-14 @336px (ablation) | Quilt-LLaVA · Quilt-VQA | Closed accuracy: 64.43 -> 69.10 | Helped |
| Use independent prompts only, without reasoning-based prompts (ablation) | Quilt-LLaVA instruction tuning · PathVQA | Closed accuracy | Mixed |
| Use reasoning-based prompts only, without independent prompts (ablation) | Quilt-LLaVA instruction tuning · PathVQA and Quilt-VQA | Open recall: 12.56 -> 17.53 | Helped |
| Combine independent and reasoning-based prompts (ablation) | Quilt-LLaVA instruction tuning · VQA benchmarks | Open recall | Mixed |
Surprising findings
- GPT-4V outperformed LLaVA-style models at reading textual artifacts in images, so the authors excluded GPT-4V from evaluation.
- Stage-1 domain alignment on Quilt-1M sometimes lowered performance on PMC-VQA-Subset, which the authors attribute to issues in that benchmark.
Limitations
- Noisy data from video frames (artifacts, faces, misaligned narration) and hand-crafted clustering rules.
- GPT-4 hallucinations propagate into the training data and model outputs; model is not reliable as a diagnostic tool.
- Dataset is English-only and limited to educational YouTube content.
Experimental setup
Datasets: Quilt-1M Quilt-Instruct PathVQA PMC-VQA
Benchmarks: Quilt-VQA PathVQA PMC-VQA-Subset Instruction Following Test Set
Metrics: relative GPT-4 score recall (open-set) accuracy (closed-set)
Baselines: LLaVA LLaVA-Med PathVQA baselines GPT-4V
Builds on: LLaVA LLaVA-Med Quilt-1M Localized Narratives
Closest papers by meaning
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.88)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.88)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.88)
- MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.87)
- Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis (NeurIPS 2024 · similarity 0.87)
- FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.