Research map / Vision-language and multimodal / Multimodal LLMs

Hyperbolic Relational Prompts for Intersectional Fairness in Medical VLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FRP, a fairness-aware relational prompt learning framework for CLIP-based medical VLMs that combines a mini-batch relational graph, a hyperbolic graph layer, and fairness-modulated adjacency to generate instance-specific prompts, evaluated on FairVLMed and Harvard-GF.

Problem

Medical vision-language models amplify demographic biases, and existing fairness methods that align distributions for single attributes suffer from intersectional blindness, where mitigating one attribute worsens another.

Approach

Reformulates prompt generation as relational reasoning: samples in a mini-batch form a graph, a hyperbolic graph layer on the Poincaré ball propagates features, and sensitive-attribute-conditioned adjacency modulates message passing so that generated prompts are fairness-aware; a variance-of-group-loss fairness term is added.

Key results

• On FairVLMed, FRP reaches 77.50% AUC with race DPD 4.14 and DEOdds 6.37, versus 75.19% AUC for MaPLe and 15.37 race DEOdds for CLIP.
• On Harvard-GF, FRP reaches 85.94% AUC, above BiomedCLIP at 83.12%, with race DEOdds 8.67.
• Ablation on FairVLMed shows removing the fairness loss raises gender DPD from 0.38 to 10.26 and removing the fairness modulator raises race DPD from 4.14 to 8.20.

Surprising findings

• Removing the fairness loss causes gender DPD to rise sharply (0.38 to 10.26) while overall AUC rises slightly (77.50 to 78.80).

Limitations

• Assumes binary gender labels and does not capture the full diversity of gender identities.
• Relational graphs are built within each mini-batch, adding computational overhead as samples or attributes grow.
• Intersectional structure is modeled as a hierarchy that may not capture complex social relationships.

Experimental setup

Datasets: FairVLMed Harvard-GF

Benchmarks: FairVLMed Harvard-GF

Metrics: AUC ES-AUC group-wise AUC DPD DEOdds

Baselines: CLIP FairCLIP CoOp CoCoOp VPT MaPLe BiomedCLIP MedCLIP PubMedCLIP

Builds on: CLIP CoOp CoCoOp MaPLe FairCLIP

Closest papers by meaning

• FairCLIP: Harnessing Fairness in Vision-Language Learning (CVPR 2024 · similarity 0.90)
• Intersectional Unfairness Discovery (ICML 2024 · similarity 0.86)
• FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs (ICLR 2025 · similarity 0.85)
• FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling (ICLR 2024 · similarity 0.85)
• FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants (CVPR 2026 · similarity 0.85)
• Intersectional Fairness via Mixed-Integer Optimization (NeurIPS 2026 · similarity 0.85)