Research map / Label-efficient and robust learning / Self-supervised and contrastive
Contextures: Representations from Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the contexture theory: many representation learning methods (supervised, SSL, manifold/graph) learn the top-d singular functions of an expectation operator induced by a context variable, proves optimality on compatible tasks, and proposes a spectrum-based task-agnostic context usefulness metric.
Problem
Lack of a systematic characterization of what representations foundation models learn, and how to judge which learning contexts are useful without knowing downstream tasks.
Approach
Representation learning is cast as learning from the association between input X and a context variable A; methods that learn the top-d singular functions of the induced expectation operator are optimal for tasks compatible with the context. A context usefulness metric built from singular value decay (moderate association is best) is proposed.
Key results
- Representations of large neural networks on abalone reach CCA up to about 0.9 and mutual KNN above 0.8 with the exact top-d eigenfunctions, with diminishing or negative gains beyond a certain width/depth.
- Across 140 contexts on 28 OpenML datasets, the proposed metric tau achieves median Pearson correlation 0.587 (median distance correlation 0.659) with actual prediction error.
- Theory shows the contexture-learning encoder minimizes worst-case linear probe error over (1-epsilon)-compatible tasks, with error (s1^2-(1-eps)^2)/(s1^2-s_{d+1}^2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the spectrum-based context usefulness metric τ to rank contexts | Linear-probe prediction error across 28 OpenML datasets · 28 OpenML datasets | Pearson correlation with actual error: 0.431 mean; 0.587 median | Mixed |
| Use distance correlation alongside τ to capture nonlinear association | Linear-probe prediction error across 28 OpenML datasets · 28 OpenML datasets | Distance correlation with actual error: 0.611 mean; 0.659 median | Helped |
| Increase K for the KNN context from 5 to 30 to 150 (ablation) | Kernel-PCA representations on abalone · abalone | Linear-probe prediction error | Mixed |
| Vary random-crop ratio from α=0.5 to α=0.2 to α=0.05 (ablation) | LeNet with non-contrastive learning and a linear probe · MNIST binary classification (digit greater than 4) | Linear-probe prediction error | Mixed |
| Increase neural-network width from narrow to sufficiently wide (ablation) | VICReg representation on abalone with KNN context (K=30) · abalone | CCA alignment with top-d eigenfunctions | Mixed |
| Increase neural-network width beyond 512 at depth 3 (ablation) | VICReg representation on abalone with KNN context (K=30) · abalone | CCA alignment with top-d eigenfunctions | Hurt |
| Increase neural-network depth beyond a sufficient depth (ablation) | VICReg representation on abalone with KNN context (K=30) · abalone | CCA alignment with top-d eigenfunctions | Hurt |
| Use width 512 instead of widths 1024 or 2048 (ablation) | VICReg representation on abalone with KNN context (K=30) · abalone | CCA alignment with top-d eigenfunctions | Helped |
| Use τ to predict error when τ is close to β + 1 | Context selection for downstream tasks · OpenML datasets | Correlation with actual prediction error | Hurt |
| Compare different types of contexts using the spectrum-only metric τ | Context selection for downstream tasks · OpenML datasets | Prediction-error ranking | Hurt |
Surprising findings
- On MNIST the strongest-association context gives the lowest error, while on abalone the weakest context is best, showing usefulness depends on the downstream task.
- Alignment with top eigenfunctions does not keep improving with model size; overly wide or deep networks show reduced alignment, attributed to optimization difficulty.
Limitations
- Analysis targets minimizers of objectives, not gradient-trained models that oscillate at the edge of stability.
- Ignores inductive bias of model architectures and assumes a fixed input distribution with no distribution shift.
- Metric fails on some datasets (e.g., negative Pearson on brazilian_houses and fifa) and cannot compare contexts of different types with similar spectra.
Experimental setup
Datasets: abalone MNIST OpenML 28 classification and regression datasets
Benchmarks: abalone MNIST 28 OpenML datasets (credit-approval, breast-w, diabetes, etc.)
Metrics: CCA mutual KNN mean squared error Pearson correlation distance correlation
Baselines: kernel PCA VICReg
Builds on: spectral contrastive loss VICReg neural collapse positive-pair kernel theory (HaoChen et al.)
Closest papers by meaning
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.86)
- Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers (NeurIPS 2026 · similarity 0.85)
- In-Context Learning with Representations: Contextual Generalization of Trained Transformers (NeurIPS 2024 · similarity 0.85)
- Scale-invariant attention (NeurIPS 2025 · similarity 0.85)
- In-Context Learning and Occam's Razor (ICML 2025 · similarity 0.85)
- Context is Environment (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.