Research map / Vision-language and multimodal / Multimodal LLMs
MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MM-CamObj, a camouflaged-scene multimodal dataset (CamObj-Align and CamObj-Instruct) and CamObj-Bench, and fine-tunes Llava-v1.5 with curriculum learning to create CamObj-Llava.
Problem
Large vision-language models perform poorly on scenes with camouflaged objects, largely because training data contains few camouflage samples.
Approach
Construct a camouflage-specific image-text alignment and instruction dataset from existing camouflage segmentation datasets using GPT-4o plus manual review, then train Llava-v1.5 with LoRA using a length-based curriculum learning strategy.
Key results
- CamObj-Llava-7B reaches 90.9% on Easy VQA, surpassing GPT-4o (88.5%) by 2.4 points.
- CamObj-Llava-13B achieves 59.4 mIoU on Bbox Location, versus 45.0 for Gemini-1.5-pro, a 14.4-point gain.
- Llava-v1.5-7B improves from 78.2 to 90.9 on Easy VQA, from 59.0 to 69.8 on Hard VQA, and from 37.4 to 56.7 on Bbox Location after training on MM-CamObj.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llava-v1.5-13B on MM-CamObj | Llava-v1.5-13B · Bbox Location | mIoU: 45.0 -> 59.4 | Helped |
| Fine-tune Llava-v1.5-13B on MM-CamObj | Llava-v1.5-13B · Image Caption | CLIPScore: 74.7 -> 85.7 | Helped |
| Fine-tune Llava-v1.5-13B on MM-CamObj | Llava-v1.5-13B · Mask Match | accuracy: 45.7 -> 50.0 | No clear effect |
| Fine-tune Llava-v1.5-13B on MM-CamObj | Llava-v1.5-13B · Mask TF | accuracy: 47.0 -> 49.5 | No clear effect |
| Train on 10% of CamObj-Align and CamObj-Instruct samples (ablation) | Llava-v1.5-7B · Easy VQA | accuracy: 78.2 -> 84.6 | Helped |
| Train on 10% of CamObj-Align and CamObj-Instruct samples (ablation) | Llava-v1.5-7B · Hard VQA | accuracy: 59.0 -> 61.7 | Helped |
| Train on 10% of CamObj-Align and CamObj-Instruct samples (ablation) | Llava-v1.5-7B · Bbox Location | mIoU: 37.4 -> 42.7 | Helped |
| Increase the training-data fraction from 10% to 20%, 50%, then 100% (ablation) | CamObj-Llava-7B · Easy VQA, Hard VQA, and Bbox Location | task scores: 84.6, 61.7, 42.7 -> 86.0, 63.9, 44.2 -> 87.8, 65.5, 49.4 -> 90.9, 69.8, 56.7 | Helped |
| Use Len_all curriculum instead of no curriculum (ablation) | Llava-v1.5-7B · Bbox Location | mIoU: 52.8 -> 56.7 | Helped |
| Use Len_ins curriculum instead of no curriculum (ablation) | Llava-v1.5-7B · Hard VQA | accuracy: 68.9 -> 67.8 | Hurt |
| Use Len_ins curriculum instead of no curriculum (ablation) | Llava-v1.5-7B · Bbox Location | mIoU: 52.8 -> 52.2 | Hurt |
| Use Len_rsp curriculum instead of no curriculum (ablation) | Llava-v1.5-7B · Hard VQA | accuracy: 68.9 -> 70.6 | Helped |
Surprising findings
- Open-source models, including CamObj-Llava, perform near random chance on multi-image Mask Match and Mask TF tasks.
- Instruction-length curriculum mode slightly hurt Hard VQA and Bbox Location relative to no curriculum.
Limitations
- Training data is small (about 11k images) and sourced only from four camouflage datasets.
- Multi-image tasks remain near random for open-source LVLMs, and training does not include multi-image inputs.
- CamObj-Llava underperforms GPT-4o and Gemini on Hard VQA.
Experimental setup
Datasets: COD-10K MoCA-Mask PlantCamo CPD-1K
Benchmarks: CamObj-Bench
Metrics: accuracy mIoU CLIPScore
Baselines: Llava-v1.5-7B Llava-v1.5-13B GPT-4o GPT-4o-mini Gemini-1.5-pro MiniCPM-V-2.5 Qwen-VL-Chat InstructBLIP Idefics Fuyu-8B Kosmos2
Builds on: Llava-v1.5 LoRA curriculum learning
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.86)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.85)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.85)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.85)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.85)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.