Research map / Label-efficient and robust learning / Recognition and long-tail
Incremental Sequence Classification with Temporal Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TC-λ, a temporal-difference-style cross-entropy loss for training incremental sequence classifiers, which improves prefix and full-sequence accuracy on text classification and early verification of LLM math solutions.
Problem
Training classifiers that make accurate predictions at every prefix of a sequence as elements are revealed, where standard direct cross-entropy uses only the final label for early prefixes and is noisy.
Approach
Enforce temporal consistency between successive predictions (the class distribution at step t equals the expected distribution at t+1) by using soft targets built from a lambda-weighted average of future predictive distributions from a reference model, a cross-entropy analogue of TD(lambda) for classification.
Key results
- On OHSUMED, TC-λ reaches 33.7% accuracy on 4-token prefixes vs 30.5% for DCE, and 81.8% on full sequences vs 81.1% for DCE.
- TC-λ matches DCE with roughly 10x larger models: DCE needs ~10x parameters to match TC-λ ROC AUC on OHSUMED.
- On GSM8K with Qwen2.5-0.5B, TC-λ is almost 70% accurate at distinguishing correct from incorrect generations after 8 tokens, and boosted best-of-N needs 23-33% fewer tokens for a given accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with TC-λ instead of DCE | OPT-125M incremental classifier · Ohsumed | accuracy: 30.5 -> 33.7 (4-token); 65.5 -> 68.3 (16-token); 81.1 -> 81.8 (all) | Helped |
| Train with TC-λ instead of DCE | OPT-125M incremental classifier · Newsgroups | accuracy: 27.7 -> 33.4 (4-token); 40.1 -> 44.7 (16-token); 89.0 -> 88.5 (all) | Mixed |
| Train with TC-λ instead of DCE | OPT-125M incremental classifier · IMDb | accuracy: 63.5 -> 64.7 (4-token); 74.7 -> 75.7 (16-token); 94.4 -> 94.9 (all) | Helped |
| Train with TC-λ instead of DCE | OPT-125M incremental classifier · AG-News | accuracy: 80.0 -> 81.4 (4-token); 92.6 -> 93.0 (16-token); 94.8 -> 95.0 (all) | Helped |
| Use temporal-consistency training rather than DCE | incremental classifier · Text classification benchmarks | accuracy | Helped |
| Replace cross-entropy with squared loss (ablation) | TC-λ/LSTD(λ) and DCE/direct ℓ₂ methods · Ohsumed and Newsgroups | accuracy | Hurt |
| Replace cross-entropy with squared loss (ablation) | TC-λ/LSTD(λ) and DCE/direct ℓ₂ methods · IMDb and AG-News | accuracy | No clear effect |
| Set λ=1, recovering DCE, instead of a λ<1 temporal-consistency setting (ablation) | TC-λ · Ohsumed | accuracy | Hurt |
| Use TC-λ verifier instead of DCE verifier | Qwen2.5-0.5B verifier · GSM8K generations after few tokens | ROC AUC | Helped |
| Use boosted best-of-N with early verifier-based stopping instead of vanilla best-of-N or majority voting | Qwen2.5-0.5B generation selection · GSM8K test-time scaling | generated tokens at a given accuracy: 23\%--33\% fewer tokens | Helped |
Surprising findings
- Optimizing temporal consistency also improves full-sequence classification accuracy, not only prefix accuracy.
- Squared-loss variants achieve similar ROC AUC to cross-entropy models on binary GSM8K verification but produce markedly worse calibrated probabilities.
Limitations
- Theoretical guarantees only hold in the tabular setting; they do not strictly apply to large parametric LLMs.
- Only models up to 1.3B parameters were systematically evaluated.
- LLM verification and test-time scaling results need further evaluation, especially against state-of-the-art methods such as speculative rejection.
Experimental setup
Datasets: OHSUMED 20 Newsgroups IMDB AG-News GSM8K
Benchmarks: OHSUMED 20 Newsgroups IMDB AG-News GSM8K
Metrics: accuracy ROC AUC negative log-likelihood KL divergence between successive predictions
Baselines: GPT-4o Most frequent Filtering Last token Specialist Direct l2 loss LSTD(lambda) DCE Majority voting Vanilla best-of-N
Builds on: temporal-difference learning TD(lambda) LSTD(lambda) OPT Qwen2.5
Closest papers by meaning
- Prediction Error-based Classification for Class-Incremental Learning (ICLR 2024 · similarity 0.86)
- Adaptive Discovering and Merging for Incremental Novel Class Discovery (AAAI 2024 · similarity 0.85)
- Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings (NeurIPS 2024 · similarity 0.85)
- Generative Multi-modal Models are Good Class Incremental Learners (CVPR 2024 · similarity 0.85)
- ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference (AAAI 2024 · similarity 0.85)
- eTag: Class-Incremental Learning via Embedding Distillation and Task-Oriented Generation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.