Research map / Vision-language and multimodal / Multimodal LLMs
Cloud-Device Collaborative Learning for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CD-CCA, a cloud-device collaborative continual adaptation framework that uses uncertainty-guided token sampling for uplink, adapter-based distillation in the cloud, and quantized dynamic weight updates for downlink to improve compressed on-device MLLMs under domain shift.
Problem
Compressed MLLMs deployed on client devices lose generalization when facing distribution shifts, while device resources limit on-device updates and cloud-device bandwidth constrains communication.
Approach
A three-part cloud-device loop: entropy and MC-dropout variance select uncertain tokens to upload (UTS), the cloud distills a large LLaMA-Adapter teacher into a small student via adapter-level query, representation and pseudo-label losses (AKD), and only quantized weight deltas are sent back (DWC).
Key results
- On VQAv2-to-AOKVQA, CD-CCA reaches a mean MC accuracy of 51.88 versus 47.95 for source-only (+3.93), with DA gain of +3.19.
- On COCO-to-nocaps, CD-CCA improves overall BLEU-4 by +1.53 and CIDEr by +2.20 over source-only.
- In real-robot tests, Uplink-UTS cuts uplink data from 31.10 MB to 65.54 KB and Downlink-DWC cuts downlink data from 14.48 GB to 0.791 MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CD-CCA rather than source-only adaptation | LLaMA-Adapter 7B · VQAv2-to-AOKVQA, MC | VQA accuracy (%): +3.93 | Helped |
| Apply CD-CCA rather than source-only adaptation | LLaMA-Adapter 7B · VQAv2-to-AOKVQA, DA | VQA accuracy (%): +3.19 | Helped |
| Apply CD-CCA rather than source-only adaptation | LLaMA-Adapter 7B · COCO-to-nocaps, All | BLEU@4: +1.53 | Helped |
| Apply CD-CCA rather than source-only adaptation | LLaMA-Adapter 7B · COCO-to-nocaps, All | CIDEr: +2.20 | Helped |
| Add pseudo-labels (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (MC): +2.53 | Helped |
| Add pseudo-labels (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (DA): +3.34 | Helped |
| Add AKD to pseudo-label training (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (MC): 50.48 -> 50.39 | Hurt |
| Add AKD to pseudo-label training (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (DA): 48.89 -> 49.05 | Helped |
| Add UTS-1 to pseudo-labels + AKD (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (MC): 50.39 -> 50.82 | Helped |
| Add UTS-1 to pseudo-labels + AKD (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (DA): 49.05 -> 48.93 | Hurt |
| Add UTS-2 to the configuration with UTS-1 (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (MC): 50.82 -> 53.19 | Helped |
| Add UTS-2 to the configuration with UTS-1 (ablation) | CD-CCA · VQAv2-to-AOKVQA | VQA accuracy (DA): 48.93 -> 49.05 | Helped |
Surprising findings
- Existing continual test-time adaptation methods such as TENT and CoTTA often degrade performance below source-only on MLLMs.
Limitations
- Evaluation is limited to two cross-domain VQA and captioning settings using LLaMA-Adapter models.
- Real-world validation uses a single robot system with Gigabit-style network assumptions and limited details.
- Core components are described with limited algorithmic detail (e.g., quantization function, selection thresholds), limiting reproducibility.
Experimental setup
Datasets: VQAv2 A-OKVQA COCO Captions 2017 nocaps COYO LAION CC3M CC12M SBU GPT4-LLM
Benchmarks: A-OKVQA nocaps VQAv2-IDS VQAv2-to-AOKVQA-IDS
Metrics: VQA accuracy BLEU-4 CIDEr
Baselines: Source-only TENT CoTTA PKD ChannelWiseDivergence
Builds on: LLaMA-Adapter TENT CoTTA PKD
Closest papers by meaning
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.86)
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.86)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.86)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.86)
- Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.