Research map / Large language models / Instruction and fine-tuning
Interweaving Memories of a Siamese Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IMSM, a model-agnostic PEFT framework that runs a siamese LLM (frozen original plus PEFT-tuned copy) and fuses their last hidden states with a query-aware gate to reduce catastrophic forgetting.
Problem
PEFT methods adapt LLMs to downstream tasks but shift parameters so the model forgets general world knowledge and reasoning (catastrophic forgetting).
Approach
Keep a frozen copy of the original LLM alongside the PEFT-tuned copy and, at each decoding step, interweave their last-layer hidden states using a low-rank, query-aware sigmoid gate computed from both models' hidden states and averaged query representations.
Key results
- On ChatGLM3-6B, IMSM over LoRA raises the four-task average from 73.10 to 75.29 (ROPES EM 74.38 to 79.15).
- On Llama3-8B, IMSM with AdaLoRA raises the average from 80.16 to 81.55 and with (IA)^3 from 78.45 to 80.05.
- On catastrophic forgetting tests (ChatGLM3, fine-tuned on ROPES), IMSM with LoRA improves the average over general benchmarks from 54.09 to 56.31; IMSM raises MultiRC accuracy relative to vanilla PEFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interweave original and fine-tuned memories using IMSM with LoRA | ChatGLM3 · ROPES | EM: 74.38 -> 79.15 | Helped |
| Interweave original and fine-tuned memories using IMSM with LoRA | ChatGLM3 · GSM8K | accuracy: 44.50 -> 48.45 | Helped |
| Interweave original and fine-tuned memories using IMSM with LoRA | Qwen1.5 · Four downstream tasks | average: 70.82 -> 72.43 | Helped |
| Interweave original and fine-tuned memories using IMSM with LoRA | Llama3 · Four downstream tasks | average: 81.43 -> 82.08 | Helped |
| Interweave original and fine-tuned memories using IMSM with AdaLoRA | ChatGLM3 · Four downstream tasks | average: 73.83 -> 75.95 | Helped |
| Interweave original and fine-tuned memories using IMSM with AdaLoRA | Qwen1.5 · Four downstream tasks | average: 71.33 -> 72.90 | Helped |
| Interweave original and fine-tuned memories using IMSM with AdaLoRA | Llama3 · Four downstream tasks | average: 80.16 -> 81.55 | Helped |
| Add query features to the gate (ablation) | IMSM + LoRA on ChatGLM3 · Four downstream tasks | average: 80.23 -> 80.65 | Helped |
| Add query features to the gate (ablation) | IMSM + (IA)^3 on ChatGLM3 · Four downstream tasks | average: 78.39 -> 79.16 | Helped |
| Add query features to the gate (ablation) | IMSM + AdaLoRA on ChatGLM3 · Four downstream tasks | average: 79.69 -> 80.98 | Helped |
| Use the learned gate instead of equally adding the hidden states (ablation) | IMSM + LoRA on ChatGLM3 · Four downstream tasks | average: 79.54 -> 80.65 | Helped |
| Use the learned gate instead of equally adding the hidden states (ablation) | IMSM + (IA)^3 on ChatGLM3 · Four downstream tasks | average: 77.82 -> 79.16 | Helped |
Surprising findings
- Inference speed drops only modestly with IMSM (e.g., LoRA on ChatGLM3 31.30 to 29.05 tokens/sec), and the authors note the extra memory as an acceptable trade-off.
- Vanilla PEFT fine-tuning sometimes lowers general benchmark accuracy (e.g., MultiRC drops from 71.74 to 66.67 with LoRA), showing forgetting.
Limitations
- Inference requires running two model copies in parallel, increasing memory use.
- Fusion is applied only to the final hidden layer; multi-layer fusion is left for future work.
- Evaluated on a limited set of tasks and 4 backbone LLMs; catastrophic forgetting assessed only on a few general benchmarks.
Experimental setup
Datasets: MRPC CoLA ROPES GSM8K WebQ FreebaseQA MultiRC
Benchmarks: MRPC CoLA ROPES GSM8K WebQ FreebaseQA MultiRC
Metrics: accuracy F1 Matthews correlation exact match tokens per second
Baselines: LoRA (IA)^3 AdaLoRA DoRA LoRAMoE prompted original LLM
Builds on: LoRA (IA)^3 AdaLoRA DoRA LoRAMoE
Closest papers by meaning
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.88)
- Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.88)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Larimar: Large Language Models with Episodic Memory Control (ICML 2024 · similarity 0.87)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.87)
- BitDelta: Your Fine-Tune May Only Be Worth One Bit (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.