Research map / Continual learning / Continual fine-tuning of LLMs
Drinking from a Firehose: Continual Learning with Web-scale Natural Language
Hexiang Hu, Ozan Şener, Fei Sha, Vladlen Koltun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConGraD adaptive steps instead of Online GD | Mixed replay · Firehose10M | word perplexity: 133.3 -> 126.5 | Improved |
| use ConGraD instead of Online GD | Online only · Firehose100M | word perplexity: 133.0 -> 125.9 | Improved |
| use ConGraD instead of Online GD | A-GEM · Firehose10M | word perplexity: 149.7 -> 148.1 | Improved |
| add user-conditioned residual adapters (ablation) | Transformer-XL · Firehose10M | word perplexity: 112.5 -> 94.1 | Improved |
| Residual Adapters instead of Multi-task Encoder (ablation) | Transformer-XL · Firehose10M | word perplexity: 100.6 -> 94.1 | Improved |
| increase validation buffer from 180K to 450K (ablation) | Mixed Replay x ConGraD · Firehose10M | word perplexity: 124.8 -> 127.3 | Hurt |
| use stratified instead of FIFO validation buffer (ablation) | Mixed Replay x ConGraD · Firehose10M | word perplexity: 127.9 -> 126.5 | Improved |
| increase Online GD steps K from 3 to 5 (ablation) | Replay only · Firehose10M | word perplexity: 158.1 -> 176.5 | Hurt |
| increase ConGraD max steps K from 3 to 5 (ablation) | Mixed replay · Firehose10M | word perplexity: 127.0 -> 126.5 | Improved |
| scale data from Firehose10M to 100M | Mixed Replay x ConGraD · test perplexity | word perplexity: 126.5 -> 115.6 | Improved |
| use ConGraD instead of Online GD on earlier benchmarks | Mixed Replay · Disjoint CIFAR | final mean accuracy: 27.6 -> 31.3 | Improved |
| use ConGraD instead of Online GD | Online only · Disjoint CIFAR | final mean accuracy: 16.0 -> 15.6 | No clear effect |
Closest papers by meaning
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.87)
- HOP to the Next Tasks and Domains for Continual Learning in NLP (AAAI 2024 · similarity 0.86)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.86)
- Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025 · similarity 0.86)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.85)
- ViLCo-Bench: VIdeo Language COntinual learning Benchmark (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
