AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

CAR-LoRA: Training Compression-Aware and Robust LoRA Adapters for Evolving LLMs

Rana Shahroz, Zhen Tan, Ruichen Zhang, Hua Wei, Tianlong Chen, Charles Fleming

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply BF16-trained LoRA naively to INT4-quantized modelLoRA adapter · MATHaccuracy: 16.5 -> 6.4Hurt
train one CAR-LoRA adapter across compressions instead of per-scheme QLoRACAR-LoRA [FP4] · MATHaccuracy: 16.1 -> 16.7No clear effect
train with layer skipping simulation and evaluate under layer skippingCAR-LoRA [LS] · GSM8Kaccuracy: 38.9 -> 31.1Hurt
train on all compression operators instead of quantization only, evaluated under layer skipping (ablation)CAR-LoRA · SQA (LS), Mistral-7Baccuracy: 53.5 -> 64.4Improved
train on all compression operators instead of pruning+LS only, evaluated at INT4 (ablation)CAR-LoRA · SQA (INT4), Mistral-7Baccuracy: 58.1 -> 68.4Improved
add quantization-aware training to BF16 LoRA, evaluated at INT4 (ablation)CAR-LoRA · SQA (INT4), Mistral-7Baccuracy: 54.3 -> 67.9Improved
use CAR-LoRA instead of standard LoRA across 4 model updatesCAR-LoRA · SQA, Llama-3.1-8Baccuracy: 54.9 -> 68.2Improved
evaluate on quantization unseen in training (FP4 withheld)CAR-LoRA · GSM8Kaccuracy: 38.49 -> 37.42Hurt
evaluate on unseen INT2 instead of INT3CAR-LoRA · SQAaccuracy: 61.7 -> 43.5Hurt
use CAR-LoRA instead of specialized QLoRA on Mistral-7B/Gemma-2-9BCAR-LoRA · SQA, Gemma-2-9B INT8accuracy: 74.5 -> 74.0Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy