Research map / Vision-language and multimodal / Multimodal LLMs
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OBI-Bench, a 5,523-image benchmark evaluating 23 LMMs on five oracle bone inscription tasks (recognition, rejoining, classification, retrieval, deciphering) under zero-shot VQA-style queries.
Problem
Whether large multimodal models have the fine-grained perception and domain knowledge needed for expert-level oracle bone inscription (OBI) processing tasks such as recognition, rejoining, classification, retrieval, and deciphering.
Approach
A task-oriented, OBI-specific VQA benchmark spanning excavation-stage to synthesis-stage OBI images (original bones, rubbings, fragments, cropped characters, handprinted), with What/Yes-or-No/How/Where question types, two newly curated datasets (O2BR and OBI-rejoin), and human baselines from untrained public participants.
Key results
- GPT-4o reaches 0.6114 What-query similarity on O2BR recognition and 0.3876 average BERTScore on deciphering, the best among proprietary LMMs and roughly on par with public-level humans (0.3872).
- GPT-4o and Gemini 1.5 Pro achieve over 76% Acc@10 on OBI rejoining (GPT-4o 78.47%).
- Open-source LMMs are far from usable: average Acc@1, Acc@5, Acc@10 of 3.85%, 11.39%, 18.65% on rejoining; Where-query mIoU for most models is near zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add role assignment and case instruction (three examples) to the deciphering pre-prompt (ablation) | Gemini 1.5 Pro, GPT-4o, Qwen-VL-Max, GLM-4V · OBI deciphering (average BERTScore) | BERTScore: small gains, e.g. GPT-4o 0.3876 to 0.3892; Qwen-VL-Max 0.2471 to 0.2681 | Helped |
| add role assignment only (no case examples) to pre-prompt (ablation) | GPT-4o · OBI deciphering (average BERTScore) | BERTScore: 0.3876 to 0.3856 | No clear effect |
| add a Yes-or-No keyword ('oracle bone') to query | most LMMs · O2BR and YinQiWenYuan recognition Yes-or-No | accuracy | Helped |
| open-source LMMs vs proprietary on rejoining | open-source LMMs · OBI rejoining How task | Acc@10: average 18.65% vs GPT-4o 78.47% | Hurt |
| larger model scale (e.g., InternVL2-Llama3-76B vs smaller) | classification with growing class count · OBI classification, 10 to 100 classes | accuracy | Helped |
| increase number of character classes from 10 to 100 | LMM classification · mixed-source OBI classification | accuracy | Hurt |
| use handprinted/cleaner images instead of noisy OBI125 rubbings | LMM classification · HWOBC/Oracle-50k vs OBI125 classification | Yes-or-No accuracy: GPT-4o surpassed by 16.4% and 19.2% on OBI125 | Helped |
| open-source LMMs compared to proprietary on deciphering | open-source LMMs · OBI deciphering | BERTScore: proprietary outperform open-source by average 145.9% | Hurt |
| query format from Yes-or-No to How probability | proprietary LMMs on rejoining · OBI rejoining | Acc@1: e.g. GPT-4o 32.21% Yes-or-No vs 29.13% Acc@1 How | Helped |
Surprising findings
- Proprietary LMMs approach or match untrained public-level humans on some deciphering scenarios, while failing badly on fine-grained localization.
- InternLM-XComposer2-VL (7B) outperforms larger InternVL2-Llama3-76B and InternVL2-40B on deciphering, attributed to training data rather than scale.
Limitations
- Human baselines are five untrained college students, not OBI experts.
- Deciphering is scored via BERTScore against dictionary-style descriptions, which lacks dedicated OBI decipherment criteria.
- Evaluation is zero-shot only; the paper reports pre-prompt effects on only four models.
Experimental setup
Datasets: YinQiWenYuan-detection HWOBC Oracle-50K OBI125 OBC306 OBI-IJDH EVOBC OBI Component 20 HUST-OBS
Benchmarks: OBI-Bench
Metrics: cosine similarity accuracy mean relative error mIoU Acc@k Recall@k mAP BERTScore
Baselines: GPT-4o GPT-4v Gemini 1.5 Pro Gemini 1.5 Flash Qwen-VL-Max GLM-4V InternVL2-Llama3-76B InternVL2-40B InternVL2-8B LLaVA-NeXT mPLUG-Owl3 MiniCPM-V 2.6
Builds on: LLaVA InternVL2 BERTScore
Closest papers by meaning
- Component-Level Segmentation for Oracle Bone Inscription Decipherment (AAAI 2025 · similarity 0.90)
- ProtoSnap: Prototype Alignment For Cuneiform Signs (ICLR 2025 · similarity 0.86)
- Making Visual Sense of Oracle Bones for You and Me (CVPR 2024 · similarity 0.86)
- LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images (CVPR 2024 · similarity 0.84)
- VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery (ICLR 2026 · similarity 0.84)
- Archaeoscape: Bringing Aerial Laser Scanning Archaeology to the Deep Learning Era (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.