AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Drinking from a Firehose: Continual Learning with Web-scale Natural Language

Hexiang Hu, Ozan Şener, Fei Sha, Vladlen Koltun

TPAMI 2022 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 23 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use ConGraD adaptive steps instead of Online GDMixed replay · Firehose10Mword perplexity: 133.3 -> 126.5Improved
use ConGraD instead of Online GDOnline only · Firehose100Mword perplexity: 133.0 -> 125.9Improved
use ConGraD instead of Online GDA-GEM · Firehose10Mword perplexity: 149.7 -> 148.1Improved
add user-conditioned residual adapters (ablation)Transformer-XL · Firehose10Mword perplexity: 112.5 -> 94.1Improved
Residual Adapters instead of Multi-task Encoder (ablation)Transformer-XL · Firehose10Mword perplexity: 100.6 -> 94.1Improved
increase validation buffer from 180K to 450K (ablation)Mixed Replay x ConGraD · Firehose10Mword perplexity: 124.8 -> 127.3Hurt
use stratified instead of FIFO validation buffer (ablation)Mixed Replay x ConGraD · Firehose10Mword perplexity: 127.9 -> 126.5Improved
increase Online GD steps K from 3 to 5 (ablation)Replay only · Firehose10Mword perplexity: 158.1 -> 176.5Hurt
increase ConGraD max steps K from 3 to 5 (ablation)Mixed replay · Firehose10Mword perplexity: 127.0 -> 126.5Improved
scale data from Firehose10M to 100MMixed Replay x ConGraD · test perplexityword perplexity: 126.5 -> 115.6Improved
use ConGraD instead of Online GD on earlier benchmarksMixed Replay · Disjoint CIFARfinal mean accuracy: 27.6 -> 31.3Improved
use ConGraD instead of Online GDOnline only · Disjoint CIFARfinal mean accuracy: 16.0 -> 15.6No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy