Research map / Vision-language and multimodal / Multimodal LLMs

Is `Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes egocentric instruction tuning (LoRA SFT on 7,335 LLaVA-style orientation QA items generated from manually annotated ImageNet images) to improve MLLM object orientation understanding, and introduces EgoOrientBench with Choose/Verify/Freeform tasks across five image domains.

Problem

MLLMs struggle to interpret object orientation, which the authors attribute to inconsistent orientation annotations in image-text training data that mix observer and object perspectives.

Approach

Standardize orientation labels to an egocentric (user/camera) eight-class scheme, generate instruction data with three response types (detail recognition, prior-knowledge reasoning, orientation alignment), and LoRA-tune MLLMs on it while keeping the visual encoder frozen.

Key results

• On EgoOrientBench Choose, LLaVA-1.5 average rises from 17.9% zero-shot to 33.7% with egocentric instruction tuning.
• InternVL2 (4B) Verify average improves from 57.7% to 61.4% and Freeform from 39.6% to 48.2%.
• Spatial reasoning accuracy improves by 15.0 points for LLaVA-1.5 (47.5 to 62.5) on COCO-Spatial, while general benchmarks stay roughly stable.

Measured effects reported in the paper

Surprising findings

• Zero-shot MLLMs score below the majority-class 'Popular' baseline on the Choose task, attributed to strong orientation bias toward Front classes.
• Improvements occur with the visual encoder frozen, suggesting limits are not solely due to the encoder, contrary to prior claims in MMVP and D3.

Limitations

• Only single-object scenarios and eight discrete orientation classes; cannot capture continuous or complex independent body/head orientations (e.g., zebra, giraffe).
• Freeform evaluation relies on GPT-4o as judge.
• Training data is limited to 2,445 ImageNet images and class imbalance exists (e.g., Back 84 samples).

Experimental setup

Datasets: ImageNet D3 DomainNet PACS OmniObject3D COCO-Spatial MME MMStar MMMU POPE

Benchmarks: EgoOrientBench MME MMStar MMMU POPE COCO-Spatial

Metrics: accuracy freeform accuracy judged by GPT-4o

Baselines: LLaVA-1.5 zero-shot mPLUG-Owl2 zero-shot InternVL2 zero-shot Gemini-1.5 Flash-8B GPT-4o Popular majority-class baseline

Builds on: LLaVA visual instruction tuning MMVP D3 benchmark

Closest papers by meaning

• EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World? (NeurIPS 2025 · similarity 0.88)
• Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.87)
• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
• ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.87)
• MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.86)
• In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.86)