Research map / Continual learning / Online and replay-based
MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoRe, a modular hierarchical latent-representation framework for sequential data with identifiability guarantees from cross-layer time-delayed dependencies, enabling continual adaptation via module reuse, alignment, and expansion.
Problem
Continual learning needs representations whose components can be selectively reused or updated without interfering with prior knowledge; existing methods make reuse/update decisions heuristically and task-specifically.
Approach
Decomposes representations into a hierarchy of fundamental-to-specific modules whose order is identified from unidirectional time-delayed cross-layer dependencies; adaptation first assesses existing modules via self-consistency scores, then uses Hard Concrete gates plus a near-identity alignment matrix to reuse, align, or expand modules while freezing the old encoder.
Key results
- On AMPS tier prediction with 10 labeled passages, MoRe_L2 (64 dims) reaches 80.0% vs 77.0% for PCA (256 dims).
- MoRe_L2 at n=50 (84.5%, 64 dims) matches PCA at n=100 (84.8%, 256 dims).
- On synthetic expansion scenarios, fine-tuning forgets up to +0.19 on C4 whereas MoRe maintains stability by isolating adaptation to new modules.
Surprising findings
- Removing the alignment matrix raises forgetting by about 0.15 across cases while preserving plasticity, identifying T as the main protector of old representations.
- On hard selective synthetic cases gate accuracy drops to 0.5, indicating ambiguity when old and new factors partially overlap.
Limitations
- Experiments use only a linear instantiation of the framework; nonlinear encoders are future work.
- Identifiability assumptions (unidirectional time-delayed structure, sufficient variability, sparsity) may only hold approximately in practice.
- Addresses one adaptation step at a time; long-horizon stability and memory for dormant modules are not studied.
Experimental setup
Datasets: synthetic sequential latent data Arxiv AMPS Wikipedia
Benchmarks: AMPS tier prediction synthetic MCC benchmarks
Metrics: mean correlation coefficient forgetting (drop in old-factor MCC) gate accuracy linear probe accuracy concept concentration
Baselines: fine-tuning scratch expansion-only reuse-only PCA
Builds on: Hard Concrete gates temporal causal representation identification JEPA
Closest papers by meaning
- A Probabilistic Framework for Modular Continual Learning (ICLR 2024 · similarity 0.88)
- Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · similarity 0.87)
- InTAct: Interval-based Task Activation Consolidation for Continual Learning (NeurIPS 2026 · similarity 0.87)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.87)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.86)
- Continual Learning with Global Alignment (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.