Research map / Label-efficient and robust learning / Self-supervised and contrastive

Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Inductive bias and minimal representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that contrastive predictive objectives with across-trajectory negatives encode slow trajectory-specific noise under fixed noise, and that sampling negatives within a trajectory (WTN) recovers the dynamical variables, with longer trajectories improving performance.

Problem

Latent predictive representation learning (JEPA-style and DySIB) can confuse slowly varying, trajectory-constant noise with the dynamical variables, so probe error degrades with noise strength and does not improve with more or longer trajectories.

Approach

Replace across-trajectory negatives in the InfoNCE predictive loss with within-trajectory negatives so that slow trajectory-constant noise is shared by positives and negatives and cannot serve as a predictive shortcut; the contrast structure is framed as an inductive bias.

Key results

• Under fixed noise, ATN probe error on the moving dot rises with noise strength toward random guessing, while WTN tracks the supervised baseline over the full noise range.
• WTN's worst-case (peak) probe error over correlation time decays approximately as a power law in trajectory length T, while ATN saturates near random guessing at all T.
• The largest WTN error occurs at intermediate noise correlation times tau approximately equal to T, forming a peak that weakens but does not vanish with longer trajectories.

Surprising findings

• Under fixed noise, WTN filters out noise across essentially all correlation times at sigma_U=1.5, though the noise is not negligible.
• Intermediate correlation times around tau approximately T remain the hardest regime for WTN even as T grows.

Limitations

• Only two low-dimensional systems with simple probe targets (dot position, pendulum angle) are studied.
• WTN requires each trajectory to cover enough phase space; stationary or periodic-orbit trajectories are problematic.
• Analysis restricted to InfoNCE-style contrastive losses; non-contrastive losses untested.

Experimental setup

Datasets: synthetic moving-dot dataset experimental pendulum video dataset

Benchmarks: synthetic moving-dot probe RMSE pendulum angle probe RMSE

Metrics: probe RMSE excess RMSE over supervised baseline

Baselines: SimCLR-predictor with across-trajectory negatives DySIB with across-trajectory negatives supervised regressor

Builds on: SimCLR-predictor DySIB InfoNCE CPC

Closest papers by meaning

• Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learning (NeurIPS 2026 · similarity 0.89)
• Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.88)
• Self-supervised contrastive learning performs non-linear system identification (ICLR 2025 · similarity 0.87)
• Identifying latent state transitions in non-linear dynamical systems (ICLR 2025 · similarity 0.87)
• Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference (NeurIPS 2024 · similarity 0.86)
• Steering Optimisation Trajectories in Diffusion Representation Learning (NeurIPS 2026 · similarity 0.85)