Research map / Vision-language and multimodal / Multimodal LLMs
Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a concept-drift-aware BLIP-style vision-language model with a T-distribution hyperspherical adapter (Thp) that mitigates long-tailed gradual drift and detects OOD sudden drift, plus a new OpenMMlo long-tailed open-world image-caption dataset.
Problem
Vision-language models suffer from gradual drift (long-tailed data) and sudden drift (OOD data) during pre-training and fine-tuning, degrading image-text alignment and OOD separability in open-world streaming settings.
Approach
Extends concept drift theory to multiple modality streams and replaces the vMF hyperspherical metric with a heavier-kurtosis T-distributed spherical metric (Thp) used as a contrastive similarity in pre-training and as a router/OOD detector (KNN on Thp distance) in fine-tuning.
Key results
- On ImageNet-LT (ViT-B/16, from scratch), the proposed model reaches 69.4% top-1 (384 resolution) vs LiVT 63.8%.
- Under only-fine-tuning with pretrained CLIP, the method reaches 77.2% on ImageNet-LT and 81.7% on iNaturalist 2018, vs LIFT 77.0% and 79.1%.
- On OOD detection trained on CIFAR100-LT, the method achieves 98.7 AUROC on SVHN (FPR 8.3) and beats CIDER on three of four datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| T-distributed (Thp) metric replacing cosine loss in long-tailed pre-training (ablation) | VL model pre-training on ImageNet-LT · ImageNet-LT image-text alignment | ID intra-class compactness (degrees, lower better): 49.2 -> 36.2 | Helped |
| T-distributed (Thp) metric replacing cosine loss in long-tailed pre-training (ablation) | VL model pre-training on ImageNet-LT · ImageNet-LT ID vs OOD separability | separability (higher better): 80.7 -> 101.3 | Helped |
| T-adapter in pre-training only (ablation) | ViT-Base VL model · ImageNet-LT | Top-1 accuracy: +2.7 (56.0 -> 58.7) | Helped |
| T-adapter in fine-tuning only (ablation) | ViT-Base VL model · ImageNet-LT | Top-1 accuracy: +9.1 (56.0 -> 65.1) | Helped |
| T-adapter in both pre-training and fine-tuning (ablation) | ViT-Base VL model · ImageNet-LT | Top-1 accuracy: +13.4 (56.0 -> 69.4) | Helped |
| fixed concentration kappa=16 vs kappa=4, 64, 128 (ablation) | T-distributed adapter · ImageNet-LT | Top-1 accuracy: 69.4 vs 67.2 (k=4), 64.9 (k=64), 61.1 (k=128) | Mixed |
| learnable concentration kappa (init 16) instead of fixed 16 (ablation) | T-distributed adapter · ImageNet-LT | Top-1 accuracy: 69.4 -> 68.3 | Hurt |
| T-distributed adapter (Thp-based) in full pipeline vs LIFT/VL-LTR fine-tuning baselines | CLIP-initialized ViT-B/16 · ImageNet-LT long-tailed classification | Top-1 accuracy (All): 77.2 vs 77.0 (LIFT), 77.2 (VL-LTR) | Helped |
| T-distributed adapter pipeline trained from scratch vs LiVT baseline | ViT-B/16 on ImageNet-LT · ImageNet-LT long-tailed classification | Top-1 accuracy (All): 67.6 vs 63.8 (LiVT†, 384 res) | Helped |
| Thp KNN OOD detection vs CIDER baseline | CIFAR100-LT trained ResNet-34 model · OOD detection SVHN/LSUN/iSUN/Texture | FPR95 and AUROC: SVHN FPR 8.3 vs CIDER 12.6; Texture FPR 45.1 vs CIDER 35.6 | Mixed |
| T-adapter pipeline vs CLIP full fine-tuning | CLIP-initialized ViT-B/16 on ImageNet-LT · ImageNet-LT | Top-1 accuracy (All): 77.2 vs 68.5 | Helped |
| Thp-based adapter as classifier for domain generalization vs DAPT | ImageNet to ImageNet-Sketch transfer · ImageNet-Sketch | Top-1 accuracy: 50.2 vs 48.3 | Helped |
Surprising findings
- Thp on imbalanced LT data yields ID vs OOD separability (101.3) exceeding the balanced cosine baseline (98.4).
- A trainable concentration parameter underperforms a fixed kappa=16.
Limitations
- Pre-training and evaluation use small-scale datasets (ImageNet-LT and iNaturalist) rather than web-scale MLLM data.
- Results are mostly on classification proxies; downstream MLLM tasks beyond classification are not evaluated.
- Trainable kappa underperforms fixed kappa and the paper gives limited theoretical justification for the Thp choice.
Experimental setup
Datasets: ImageNet-LT ImageNet iNaturalist 2018 Places-LT CIFAR100-LT ImageNet-Sketch SVHN Places365 LSUN iSUN Texture
Benchmarks: ImageNet-LT iNaturalist 2018 SVHN LSUN iSUN Texture ImageNet-Sketch
Metrics: top-1 accuracy FPR95 AUROC intra-class compactness (cosine degree) inter-class separability
Baselines: cRT LWS MiSLAS RIDE PaCo LiVT CLIP LIFT VL-LTR CIDER CoOp VPT
Builds on: BLIP ALBEF CIDER vMF distribution t-SNE
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.89)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.88)
- IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.