Research map / Vision-language and multimodal / Multimodal LLMs
DigitalLLaVA: Incorporating Digital Cognition Capability for Physical World Comprehension in Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DigitalLLaVA, which fine-tunes LLaVA-v1.5-7B's image encoder via contrastive carrier mapping and adds three float heads (sign, exponent, fraction) to predict 32-bit floats for precise digital readings from physical instruments.
Problem
Multimodal LLMs cannot accurately read physical digits (e.g., gauges, thermometers, vernier calipers), since they encode numbers token-by-token or via regression and lack understanding of digital carriers.
Approach
Two-step injection: contrastive fine-tuning of the CLIP image encoder to recognize digit carriers, then a float32 simulation where a special [Float] token routes to three heads predicting sign, 8-bit exponent and 23-bit fraction as binary/regression targets.
Key results
- On the digit-reading datasets, DigitalLLaVA reaches 80-87% accuracy on question answering, comprehension/reasoning and multi-turn tasks, versus 3-55% for baselines such as P10 and GPT-4V.
- On precision tests, the full method achieves MAE 0.15 and RMSE 0.07 (normalized) and 84.9% accuracy, compared to 0.58/0.64 and 39.1% for LLaVA.
- On general benchmarks, DigitalLLaVA matches LLaVA-v1.5-7b (e.g., MathVista 32.2 vs 25.6, POPE 85.8 vs 86.1, MME 1757.7 vs 1808.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard number generation with 32-bit float simulation (ablation) | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 70.6 | Helped |
| add digital carrier mapping and float32 simulation together | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 84.9 | Helped |
| add digital carrier mapping and predict digits sequentially (ablation) | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 50.0 | Helped |
| replace number encoding with direct MSE regression (ablation) | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 43.3 | Helped |
| use P10 positional encoding (ablation) | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 49.8 | Helped |
| use P1000 positional encoding (ablation) | LLaVA-v1.5-7b · Digital carrier datasets | accuracy: 39.1 -> 49.5 | Helped |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · MathVista | accuracy: 25.6 -> 32.2 | Helped |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · POPE | accuracy: 86.1 -> 85.8 | Hurt |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · MME | score: 1808.4 -> 1757.7 | Hurt |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · ScienceQA | accuracy: 69.2 -> 70.8 | Helped |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · RealWorldQA | accuracy: 54.8 -> 58.3 | Helped |
| add digital carrier mapping and float32 simulation | LLaVA-v1.5-7b · MMBench | accuracy: 59.1 -> 59.3 | Helped |
Surprising findings
- The MSE regression variant overfit: it performed well on the training set but poorly on the test set.
- Float32 simulation did not degrade object recognition; carrier recognition F1 rose from 83.3 to 97.0 over LLaVA.
Limitations
- Failure cases on stopwatch, protractor and thermometer readings show errors in fine-grained recognition and image compression effects.
- Images are fixed at 256x256 resolution on synthetic-style instrument datasets, limiting evidence for real-world generality.
- Evaluation is limited to ten instrument types in a single multi-task training run.
Experimental setup
Datasets: Time Clock Speedometer Vernier Caliper Thermometer Graduated Cylinder Protractor Spring Scale Pressure Gauge Stopwatch Ammeter-Voltmeter MathVista ScienceQA
Benchmarks: MathVista ScienceQA POPE RealWorldQA MME MMBench Physical digit datasets (ten instrument datasets)
Metrics: accuracy MAE RMSE accuracy within tolerance precision recall F1-score
Baselines: Finetune LoRA GPT-4V Gemini Pro LLaVA OpenFlamingo-v2 MiniGPT-4-v2 P10 P1000 UReader XVAL
Builds on: LLaVA-v1.5 CLIP P10 / P1000 digit encoding LoRA
Closest papers by meaning
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.86)
- MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? (ICLR 2025 · similarity 0.86)
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (ICML 2025 · similarity 0.85)
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning (AAAI 2025 · similarity 0.85)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.85)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.