Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
$\boldsymbol{\lambda}$-Orthogonality Regularization for Compatible Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes λ-Orthogonality regularization, a relaxed orthogonality penalty on an affine backward adapter, plus supervised contrastive loss and a distance-based backfilling order, to make independently trained embedding models compatible for retrieval.
Problem
Backward and forward compatibility between independently trained embedding models for retrieval, so that model updates avoid costly gallery backfilling while preserving the new model's representation quality.
Approach
Replace strict orthogonality with a sigmoid-gated penalty on ||WW^T - I||_F that only activates beyond a threshold λ, letting an affine backward adapter adapt to downstream data while staying near-orthogonal; combine with forward adapter, supervised contrastive loss, and a backfilling order based on distance from class means.
Key results
- On CUB200 with ResNet-18 old and ViT-L-16 new, B_λ(φ_new)/B_λ(φ_new) reaches 75.44 CMC-Top1 (+3.66 over independent training 71.78) with ZS gain +0.028 on ImageNet1K.
- In the extending-classes ImageNet1K setting, the proposed method achieves 61.20 CMC-Top1 and 36.46 mAP on the backfilling metric, versus 60.49/35.59 for FastFill.
- Strict orthogonal adapters (B_⊥) do not improve over the independently trained new model on downstream tasks, while λ=12 gives the best trade-off between downstream adaptation and ImageNet1K preservation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use λ-orthogonality with λ=12 instead of strict orthogonality for the backward adapter (ablation) | ResNet-18 and ViT-L-16 representations · CUB, Bλ(φnew)/F(φold) | CMC-Top1: 66.89 -> 70.72 | Helped |
| Use λ=12 instead of no regularization (λ=∞) (ablation) | ResNet-18 and ViT-L-16 backward adapter · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 78.89 -> 75.44 | Hurt |
| Increase λ from 0 to 12 (ablation) | ResNet-18 and ViT-L-16 backward adapter · ImageNet1K zero-shot | CMC-Top1 gain: –0.001 -> +0.028 | Helped |
| Increase λ from 12 to 36 (ablation) | ResNet-18 and ViT-L-16 backward adapter · ImageNet1K zero-shot | CMC-Top1 gain: +0.028 -> –3.008 | Hurt |
| Use λ-orthogonality at λ=12 rather than Soft Orthogonality (w=1) (ablation) | ResNet-18 and ViT-L-16 backward adapter · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 71.54 -> 75.44 | Helped |
| Use λ-orthogonality at λ=12 rather than SRIP (w=1) (ablation) | ResNet-18 and ViT-L-16 backward adapter · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 71.66 -> 75.44 | Helped |
| Reduce the λ-orthogonality loss weight from 1 to 10⁻¹ (ablation) | ResNet-18 and ViT-L-16 backward adapter · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 75.44 -> 75.20 | No clear effect |
| Remove λ-orthogonality regularization (ablation) | ResNet-18 and ViT-L-16 backward adapter · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 71.78 -> 78.89 | Helped |
| Use distance-from-class-mean ordering for gallery backfilling instead of FastFill ordering | ResNet-34 retrieval system · Extended Classes, partial backfilling | CMC-Top1: 60.49 -> 61.20 | Helped |
| Use distance-from-class-mean ordering for gallery backfilling instead of FastFill ordering | ResNet-18 and ViT-L-16 retrieval system · Independently pretrained, partial backfilling | mAP: 55.34 -> 57.72 | Helped |
| Add λ-orthogonality-regularized backward mapping to the independently trained new model | ResNet-18 and ViT-L-16 · CUB, Bλ(φnew)/Bλ(φnew) | CMC-Top1: 71.78 -> 75.44 | Helped |
| Add a forward adapter with contrastive alignment rather than using FCT | ResNet-18 old model and CLIP new model · ImageNet1K, F(φold)/φold | CMC-Top1: 0.04 -> 61.13 | Helped |
Surprising findings
- Strict orthogonal transformations do not improve downstream performance over the independently trained new model, and soft orthogonality (λ=0) is outperformed by strict orthogonality.
- FastFill and FCT can fail to be backward compatible with the old model (e.g., FCT's F(φ_old)/φ_old near 0.04 CMC-Top1 on ImageNet1K in ViT setting is inherent to independent models), and CLIP pretrained on CC12M underperforms ResNet-18 on ImageNet1K, causing FCT to degrade the system.
Limitations
- Gains on the downstream task come at a cost in zero-shot ImageNet1K performance when λ is large or regularization is removed.
- The regularization weight must be set to w=1 to be effective; smaller weights make the constraint unenforced, and λ must be tuned per setting (λ=12 chosen empirically).
Experimental setup
Datasets: ImageNet1K CIFAR100 CUB200 Places365 Flowers102
Benchmarks: ImageNet1K CUB200 CIFAR100 Places365 Flowers102
Metrics: CMC-Top1 mAP zero-shot CMC-Top1 gain
Baselines: FCT FastFill Independent training (Ind. Train.) Soft Orthogonality (SO) SRIP
Builds on: Backward Compatible Training Forward Compatible Training FastFill soft orthogonality regularization
Closest papers by meaning
- Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (ICML 2025 · similarity 0.84)
- Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements (CVPR 2024 · similarity 0.83)
- Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation (NeurIPS 2024 · similarity 0.83)
- QuARI: Query Adaptive Retrieval Improvement (NeurIPS 2025 · similarity 0.83)
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.83)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.