Research map / Architectures and efficiency / RNNs and normalization
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Sparse Representation (CSR), a lightweight TopK sparse autoencoder with a non-negative contrastive loss placed on frozen pretrained embeddings to yield adaptive, sparse embeddings that beat Matryoshka Representation Learning in accuracy, retrieval speed, and training cost.
Problem
Matryoshka Representation Learning (MRL) for adaptive-length embeddings requires full backbone retraining and degrades noticeably at short embedding lengths, while retrieval cost and training cost remain high.
Approach
Instead of truncating dense embeddings, sparsify frozen pretrained embeddings into a wider (h=4d) latent space keeping only the top-k activations, trained with a TopK sparse autoencoder reconstruction loss plus a non-negative contrastive loss; retrieval uses sparse matrix multiplication.
Key results
- On ImageNet 1-NN at active dim 8, CSR reaches 67.78% versus 62.19% for MRL with ResNet-50 (73.84% with the SOTA RN50 backbone).
- On MTEB text tasks with NV-Embed-V2 at k=32, CSR reaches 91.02% MTOP intent-style classification accuracy on TweetSentiment and 38.06 NDCG@10 on NFCorpus, close to or above full representation on several datasets.
- Paper claims a 69x speedup on ImageNet 1-NN retrieval versus quantization-based approaches and a 61x speedup versus NV-Embed-V2 at matched performance on text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the sparse contrastive loss to SAE (ablation) | CSR with NV-Embed-V2 · Banking77 | Top-1 accuracy (%): 88.11 -> 91.02 | Helped |
| Add the sparse contrastive loss to SAE (ablation) | CSR with NV-Embed-V2 · MTOPIntent | Top-1 accuracy (%): 87.43 -> 89.86 | Helped |
| Add the sparse contrastive loss to SAE (ablation) | CSR with NV-Embed-V2 · TweetSentiment | Top-1 accuracy (%): 75.19 -> 78.55 | Helped |
| Add the sparse contrastive loss to SAE (ablation) | CSR with ResNet50 · ImageNet-1K | 1-NN accuracy: 68.91 -> 70.15 | Helped |
| Add the sparse contrastive loss to SAE (ablation) | CSR with ResNet50 · ImageNet-1K | Top-1 accuracy: 75.28 -> 75.44 | Helped |
| Use CSR rather than MRL at active dim 32 | ImageNet-1K classification · ImageNet-1K | 1-NN accuracy: 69.46 -> 70.15 | Helped |
| Use CSR rather than MRL at active dim 32 | ImageNet-1K classification · ImageNet-1K | Top-1 accuracy: 75.03 -> 75.44 | Helped |
| Use CSR rather than MRL at active dim 64 | ViT-B-16 multimodal retrieval · MS COCO and Flickr30K, in-distribution and zero-shot | Recall@5 (%): 62.61 -> 62.75; 72.43 -> 78.10; 74.22 -> 76.44; 84.79 -> 88.50; 47.47 -> 48.61; 54.42 -> 61.90; 71.16 -> 73.04; 79.00 -> | Helped |
| Increase hidden dimension up to h=4d (ablation) | CSR image representations · ImageNet | 1-NN accuracy | Helped |
| Increase hidden dimension beyond h=4d (ablation) | CSR image representations · ImageNet | 1-NN accuracy | Hurt |
| Increase input embedding dimension at equal sparsity (ablation) | CSR image representations · ImageNet | 1-NN accuracy | Helped |
| Use non-negative contrastive loss instead of only auxiliary and multi-TopK losses (ablation) | CSR sparse autoencoder · ImageNet, especially k=8, 16, 32 | active latent dimensions | Helped |
Surprising findings
- Retrieval time for sparse CSR embeddings decreases as the hidden dimension h grows, while dense MRL retrieval gets slower with higher dimension.
- Multimodal CSR still suffers from dead latents at high sparsity, and its advantage over MRL narrows as sparsity becomes stricter.
Limitations
- Dead latents persist at high sparsity, especially in multimodal image-text alignment space.
- Gap to MRL narrows under stringent sparsity in multimodal settings.
- Speed comparisons depend on backend implementation (cuBLAS vs cuSPARSE) and are reported on GPU only.
Experimental setup
Datasets: ImageNet-1K MTOPIntent Banking77 TweetSentimentExtraction BiorxivClusteringS2S BiorxivClusteringP2P TwentyNewsgroupsClustering FiQA2018 NFCorpus SciFACT MS COCO Flickr30K
Benchmarks: ImageNet-1K 1-NN ImageNet-1K Top-1 linear probe MTEB classification/clustering/retrieval subsets MS COCO retrieval Flickr30K retrieval
Metrics: 1-NN accuracy Top-1 accuracy NDCG@10 Recall@5 retrieval time
Baselines: MRL MRL-E SVD Rand-LP Rand-FS Stella-1.5B-v5 Jina-V3 Nomic-Embed-V1.5 Gecko-Embed-004 Text-Embed-3-L Arctic-Embed-L-V2 Potion-Base-2M
Builds on: Matryoshka Representation Learning TopK sparse autoencoder non-negative contrastive learning
Closest papers by meaning
- From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit (NeurIPS 2025 · similarity 0.89)
- Learning Multi-Level Features with Matryoshka Sparse Autoencoders (ICML 2025 · similarity 0.88)
- Interpreting CLIP with Hierarchical Sparse Autoencoders (ICML 2025 · similarity 0.87)
- CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation (AAAI 2025 · similarity 0.87)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.87)
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.