Research map / Vision-language and multimodal / Multimodal LLMs

MegaCoin: Enhancing Medium-Grained Color Perception for Vision-Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MegaCoin, a human-annotated dataset of 220k real images (CIFAR10, TinyImageNet, ImageNet val) with foreground color, background color, and physical environment labels, providing an SFT set, a tiered QA benchmark, and a domain generalization testbed.

Problem

VLMs have weak fine-grained color and physical-environment perception, and existing datasets lack human-labeled contextual attributes for SFT, evaluation, and domain generalization benchmarking.

Approach

Medium-grained human annotations (foreground color, background color, environment) on real images are turned into instruction pairs for SFT and a Tiered-MQA benchmark (Least/More/Most information levels), and the same labels define spurious-attribute subgroups for DG benchmarking.

Key results

• Fine-tuned LLaVA-1.5-13B (+C10&TI) outperforms GPT-4o on MegaCoin-Bench in many splits, e.g., LEAST TIFGD rises from 46.54% to 65.11% with +TI.
• On MegaCoin-Bench, GPT-4o scores as low as 26.08% on TIENV LEAST and 30.19% on C10ENV LEAST, showing weak basic perception.
• In linear probing DG benchmarking, MMD achieves the highest average accuracy (89.06%), with ERM and CORAL at 89.04%; CORAL is best on worst-group accuracy (86.01%).

Surprising findings

• GroupDRO, a subgroup-robust method, is not among the top performers in the linear probing setting, while it is top with ResNet18.
• Fine-tuning on MegaCoin-Instruct yields mostly flat or improved results on general benchmarks (MMBench, MME, VQAv2, GQA) despite the focus on basic perception.

Limitations

• SFT data uses template-based instruction pairs rather than naturally generated language.
• Bunny-1.1 underperforms GPT-4o on C10ENV and shows weaker generalization to ImageNet, which was not in its training set.
• Annotation is limited to a predefined eleven-color palette and six environment categories.

Experimental setup

Datasets: CIFAR10 TinyImageNet ImageNet validation set MMBench MME VQAv2 GQA Waterbirds CelebA MetaShift Spawrious

Benchmarks: MegaCoin-Bench MMBench MME VQAv2 GQA

Metrics: accuracy worst-group accuracy average accuracy standard deviation across seeds

Baselines: GPT-4o LLaVA-1.5-13B Bunny-1.1-8B ERM GroupDRO CORAL MMD EQRM SelfReg VREx

Builds on: DomainBed LoRA

Closest papers by meaning

• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.88)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
• Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.87)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.87)