Research map / Vision-language and multimodal / Multimodal LLMs
INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance
Chenwei Lin, Hanjia Lyu, Xian Xu, Jiebo Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add insurance-related information to the prompt (crop growth stage) (ablation) | GPT-4V · Crop growth stage detection | accuracy (%): 22.00 -> 52.00 | Improved |
| add insurance-related information to the prompt (house damage type) (ablation) | GPT-4o · House damage type detection | accuracy (%): 48.00 -> 57.00 | Improved |
| add insurance-related information to the prompt (vehicle damage severity) (ablation) | Qwen-VL-Max · Vehicle damage severity detection | accuracy (%): 72.00 -> 61.00 | Hurt |
| add insurance-related information to the prompt (vehicle damage severity) (ablation) | Qwen-VL-Plus · Vehicle damage severity detection | accuracy (%): 68.00 -> 58.00 | Hurt |
| add insurance-related information to the prompt (vehicle damage severity) (ablation) | Gemini 1.5 Flash · Vehicle damage severity detection | accuracy (%): 68.00 -> 68.00 | No clear effect |
| use distractors from Claude 3.5 Haiku or DeepSeek-V3 instead of GPT-4o (ablation) | INS-MMBench multiple-choice questions · three tasks, 1,500 questions | accuracy | No clear effect |
| scenario task multi-step reasoning vs fundamental tasks | GPT-4o, Qwen-VL-Max, Gemini 1.5 Flash · INS-MMBench scenario tasks | accuracy | Hurt |
| open-source Qwen-2.5-VL-32B vs closed-source models | Qwen-2.5-VL-32B · INS-MMBench fundamental tasks | average accuracy: 64.10 vs 69.70 (GPT-4o) | Mixed |
Closest papers by meaning
- MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.84)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.83)
- Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.83)
- FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation (AAAI 2026 · similarity 0.83)
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.82)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
