Research map / Vision-language and multimodal / Multimodal LLMs

FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FairLLaVA, a LoRA-based fine-tuning method that minimizes mutual information between pooled hidden states and demographic attributes (via a variational classifier) to reduce performance gaps of MLLMs in radiology report generation and dermoscopy QA.

Problem

Medical multimodal LLMs show uneven report-generation quality across age, race, and gender groups; standard fairness methods (reweighting, resampling, adversarial debiasing) either degrade overall performance or are costly, and fairness metrics for generative text are underdeveloped.

Approach

Add a mutual-information minimization regularizer, using a lightweight variational demographic attribute classifier whose negatives come from the mini-batch, applied to pooled middle-layer LLM hidden states during LoRA-based visual instruction tuning, plus an extension of the equity-scaled metric to generative text.

Key results

• On MIMIC-CXR, FairLLaVA-All wins 7 of 12 best ES scores, e.g., ES-BLEU-1 of 13.36 for Race vs 8.71 for the best baseline Resampling-All.
• On PadChest, FairLLaVA-All reaches the best overall BLEU-1 (25.11), BLEU-4 (12.03), RadGraph-F1 (15.66) and GREEN (40.03) among compared methods.
• Equity-scaled CheXpert-14 F1 on MIMIC-CXR is 69.21 for Race vs 24.06 for Chen et al.

Surprising findings

• The adversarial-classifier debiasing baseline strongly degrades overall clinical quality (overall GREEN drops to 9.36 vs 34.32 for FairLLaVA), suggesting catastrophic forgetting.
• Low-fairness-gap general MLLMs (e.g., Qwen2.5-7B) appear fair mainly because they perform uniformly poorly, so gap-only metrics can be misleading.

Limitations

• Requires demographic labels for training and for computing fairness gaps, which clinical datasets may lack or have unreliably.
• Targets group-level disparities; individual/cross-sectional fairness is only secondarily evaluated.
• Equity-scaled metrics have high variance (wide bootstrap confidence intervals), and hyperparameter lambdas are chosen with reference to baseline gap ratios.

Experimental setup

Datasets: MIMIC-CXR PadChest HAM10000 CheXpert NIH ChestX-ray14

Benchmarks: MIMIC-CXR PadChest HAM10000

Metrics: ES-BLEU-1 ES-BLEU-4 ES-RadGraph-F1 ES-GREEN ES-ROUGE-L ES-Acc BLEU-1 BLEU-4 RadGraph-F1 GREEN

Baselines: LLaVA LLaVA-Rad MedGemma-4B MedGemma-27B Qwen2.5-7B DeepSeek-VL2 CheXagent Reweighting Resampling Adversarial MLP classifier Chen et al.

Builds on: LLaVA LoRA BioMedCLIP Vicuna equity-scaled metric

Closest papers by meaning

• FairCLIP: Harnessing Fairness in Vision-Language Learning (CVPR 2024 · similarity 0.90)
• FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis (ICLR 2024 · similarity 0.88)
• On Fairness of Unified Multimodal Large Language Model for Image Generation (NeurIPS 2025 · similarity 0.87)
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models (ICLR 2025 · similarity 0.87)
• FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.86)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.86)