Research map / Large language models / Instruction and fine-tuning

Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Drag-and-Drop LLMs (DnD), a prompt-conditioned hyper-convolutional parameter generator that maps unlabeled task prompts directly to LoRA weights for Qwen2.5 models without per-task training.

Problem

PEFT methods like LoRA still require a separate optimization run per downstream dataset, which is costly at scale; the goal is to generate task-specific LoRA weights directly from task prompts without gradient-based tuning on the target task.

Approach

A frozen Sentence-BERT encoder embeds batches of unlabeled prompts, and a cascaded hyper-convolutional decoder is trained with MSE on tokenized LoRA checkpoints paired with prompt batches, so it can produce LoRA matrices for unseen datasets in one forward pass.

Key results

• On unseen common-sense test sets, DnD averages 21.0 points above training LoRAs (e.g., ARC-c 51.6 vs 39.5 average), and reaches 66.3 on gsm8K vs 42.9 for training LoRAs.
• On HumanEval with Qwen2.5-1.5B, DnD improves pass@1 by 15.1, pass@5 by 26.7, and pass@10 by 30.9 over the average training LoRA.
• DnD generates parameters in about 0.11 to 0.73 seconds per model on one A100, with the paper reporting up to 12,000x lower overhead than full fine-tuning.

Measured effects reported in the paper

Surprising findings

• Using answers as the condition instead of prompts yields poor results (27.0 vs 51.6 accuracy on ARC-c), attributed to low answer diversity.
• Qwen2.5-7B as the condition extractor failed to train effectively, with the paper attributing this to its heaviness and decoder-only design.

Limitations

• Training relies on many pre-collected LoRA checkpoints per task, and performance drops sharply when fewer training datasets are used (2-5 arrangement gives near-zero improvement).
• Scaling to larger models (7B-70B) is stated as an open challenge; experiments are limited to Qwen2.5 0.5B-7B and one VL model.
• Some gains come from comparisons against weak training LoRAs and the base model, and the paper is inconsistent on batch/prompt-length settings across tables.

Experimental setup

Datasets: ARC-e ARC-c BoolQ OBQA HellaSwag PIQA WinoGrande Evol-Instruct-68K-V1 Glaive-Assistant-V2 Python-Codes-25K Code-74k-ShareGPT Rosetta-Code

Benchmarks: ARC-e OBQA ARC-c PIQA HellaSwag BoolQ WinoGrande HumanEval gsm8k MATH Math-Vision Math-Vista

Metrics: accuracy pass@k improvement over training LoRAs wall-clock time memory cost

Baselines: training LoRAs RPG Qwen2.5 foundation model few-shot tuning in-context learning full-shot LoRA tuning

Builds on: LoRA RPG p-diff Sentence-BERT

Closest papers by meaning

• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.90)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
• Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.89)
• LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.89)
• AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.89)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.89)