Research map / Label-efficient and robust learning / Recognition and long-tail
Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that stationary representations from a d-Simplex fixed classifier optimally approximate backward compatibility, and proposes d-Simplex-HOC (cross-entropy plus infoNCE) for asynchronous model replacement in lifelong learning on CIFAR100R/CIFAR10.
Problem
Learning compatible representations so that updated models can be swapped in without reprocessing gallery images, which lacks a comprehensive theoretical understanding.
Approach
Proves that stationary representations learned with a d-Simplex fixed classifier optimally approximate the two compatibility inequality constraints in expectation, and adds an infoNCE term to cross-entropy (HOC loss) to capture higher-order alignment between old and new models.
Key results
- d-Simplex-HOC achieves AC 0.95 and AA7 68.13 on the 7-task sequence, versus 0.05 AC for d-Simplex-FD and near-zero AC for BCT-ER and CVS.
- On the 31-task sequence, d-Simplex-HOC reaches AC 0.65 and AA31 67.40, while all other compared methods fail to achieve compatibility.
- d-Simplex-HOC keeps improving performance gradually after model replacements with ResNet18, SENet18, and RegNetY 400MF, whereas d-Simplex-FD plateaus around task 20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use d-Simplex-HOC instead of d-Simplex-FD | Stationary representation learning · CIFAR100R/10, 7 tasks | AC: 0.05 -> 0.95 | Helped |
| Use d-Simplex-HOC instead of d-Simplex-FD | Stationary representation learning · CIFAR100R/10, 7 tasks | AA7: 56.58 -> 68.13 | Helped |
| Use d-Simplex-HOC instead of d-Simplex-FD | Stationary representation learning · CIFAR100R/10, 31 tasks | AC: 0.21 -> 0.65 | Helped |
| Use d-Simplex-HOC instead of d-Simplex-FD | Stationary representation learning · CIFAR100R/10, 31 tasks | AA31: 56.27 -> 67.40 | Helped |
Surprising findings
- Baseline compatibility methods such as BCT-ER and CVS show near-zero cross-test accuracy after model replacement, indicating non-compatible representations despite their design.
- Under the reported setup, d-Simplex-HOC is compatible in all but one instance in the 7-task sequence, which is the only exception.
Limitations
- Theorem relies on the unconstrained feature model assumption (sufficient backbone expressiveness) and a d-Simplex fixed classifier.
- Experiments are on small-scale CIFAR/ImageNet32 benchmarks with ResNet18-class models.
- Closed-form expected distances are not available for the general case, so the optimality result is shown in expectation under assumptions.
Experimental setup
Datasets: ImageNet32 CIFAR100R CIFAR10 MNIST
Benchmarks: CIFAR10 (gallery and query, open-set 1:N search)
Metrics: Average Compatibility (AC) Average multi-model Accuracy (AAt)
Baselines: FAN CVS d-Simplex-FD BCT-ER LCE-ER AdvBCT-ER Experience Replay
Builds on: Backward Compatible Training d-Simplex fixed classifier Neural Collapse InfoNCE
Closest papers by meaning
- Forward Compatible Training for Large-Scale Embedding Retrieval Systems (CVPR 2022 · similarity 0.87)
- Instance-Conditional Timescales of Decay for Non-Stationary Learning (AAAI 2024 · similarity 0.85)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.85)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.85)
- Conditional Representation Learning for Customized Tasks (NeurIPS 2025 · similarity 0.84)
- Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.