Research map / Architectures and efficiency / Pruning and efficient tuning
EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMLoC, which fine-tunes LoRA on an SVD-compressed emulator built from a small downstream calibration set, then corrects and merges the LoRA into the full model so fine-tuning fits the inference memory budget.
Problem
Fine-tuning large foundation models needs far more memory than inference, so users with inference-capable hardware often cannot fine-tune the same model; existing parameter-reduction approaches need costly continual pretraining or restrict which weights can be tuned.
Approach
Build a task-specific low-rank emulator via activation-aware SVD (SVD-LLM) on a small downstream calibration set, train standard LoRA on the emulator, then correct the LoRA weights in an orthonormal input basis so they can be merged into the original full model with compensated misalignment.
Key results
- At 25% compression, EMLoC reaches 51.6 on PMC-VQA, 79.6 on WebSRC and 46.2 on WC-VQA, beating Offsite-tuning and UPop under matched trainable parameters.
- EMLoC fine-tunes InternVL2.5-38B (95GB for full fine-tuning) within a 24GB GPU, raising WC-VQA from 51 zero-shot to 52.6.
- Ablation shows activation-aware SVD contributes most of the gain (WebSRC 74.4 to 79.0 at 25%), and LoRA correction adds a further +0.6 on WebSRC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMLoC instead of UPop | InternVL2.5-8B fine-tuning · WebSRC, 50% | accuracy: 76.4 -> 85.2 | Helped |
| Use EMLoC instead of UPop | InternVL2.5-8B fine-tuning · WC-VQA, 25% | accuracy: 44.1 -> 46.2 | Helped |
| Add LoRA correction (ablation) | EMLoC with activation-aware SVD · PMC-VQA, 25% | accuracy: 51.5 -> 51.6 | Helped |
| Add LoRA correction (ablation) | EMLoC with activation-aware SVD · WebSRC, 25% | accuracy: 79.0 -> 79.6 | Helped |
| Add LoRA correction (ablation) | EMLoC with activation-aware SVD · WC-VQA, 25% | accuracy: 45.8 -> 46.2 | Helped |
| Use activation-aware SVD instead of standard SVD (ablation) | EMLoC without LoRA correction · WebSRC, 25% | accuracy: 74.4 -> 79.0 | Helped |
| Use activation-aware SVD instead of standard SVD (ablation) | EMLoC without LoRA correction · WC-VQA, 25% | accuracy: 44.7 -> 45.8 | Helped |
| Use activation-aware SVD instead of standard SVD (ablation) | EMLoC without LoRA correction · PMC-VQA, 25% | accuracy: 51.0 -> 51.5 | Helped |
| Transfer emulator-trained LoRA to the original model | EMLoC · ChartQA, 25% | accuracy: 19.3 -> 84.2 | Helped |
| Fine-tune with EMLoC rather than zero-shot | InternVL2.5-8B · WebSRC | accuracy: 76.5 -> 85.2 | Helped |
| Apply EMLoC fine-tuning | InternVL2.5-38B · PMC-VQA | accuracy: 52.5 -> 57.0 | Helped |
| Apply EMLoC fine-tuning | InternVL2.5-38B · WebSRC | accuracy: 79.0 -> 82.1 | Helped |
Surprising findings
- The emulator alone performs very poorly (e.g., 6.3 on DocVQA at 25%), yet LoRA trained on it greatly improves the full model after transfer.
- Naive quantized QLoRA on InternVL2.5-26B/38B underperforms the half-precision zero-shot baseline.
Limitations
- Relies on off-the-shelf SVD methods designed to preserve inference behavior rather than fine-tuning dynamics, so the emulator may not capture training characteristics.
- The LoRA correction is tailored to linear LoRA modules and has no straightforward extension to other PEFT methods such as adapters.
- Large-model experiments are evaluated on a subset of the test set due to compute limits.
Experimental setup
Datasets: ChartQA DocVQA InfoVQA TextVQA PMC-VQA WebSRC WC-VQA MathQA GSM8K LAION-COCO DreamBooth
Benchmarks: ChartQA DocVQA InfoVQA TextVQA PMC-VQA WebSRC WC-VQA MathQA GSM8K
Metrics: accuracy DINO CLIP-I CLIP-T peak memory (GB) wall-clock time
Baselines: Offsite-tuning UPop LORAM InternVL 4B InternVL 2B QLoRA Houlsby Adapter
Builds on: SVD-LLM LoRA LORAM Offsite-tuning DreamBooth
Closest papers by meaning
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.86)
- Sparse High Rank Adapters (NeurIPS 2024 · similarity 0.85)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.85)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.85)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.85)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.