Research map / Label-efficient and robust learning / Self-supervised and contrastive

Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes V-pretraining, which uses a small set of verifiable downstream examples to train a lightweight task designer that constructs soft targets (language) or views/masks (vision) for self-supervised continued pretraining, while the learner is updated only by the self-supervised loss.

Problem

Continued pretraining optimizes a fixed self-supervised proxy task but is selected by downstream performance, creating a coarse feedback loop; the question is whether downstream examples can give step-level feedback without directly supervising the learner.

Approach

A task designer shapes the self-supervised targets or views and is trained to maximize the gradient alignment between the downstream loss gradient and the pretraining gradient (a first-order one-step downstream improvement proxy); the learner only receives detached self-supervised updates.

Key results

• Qwen2.5-0.5B GSM8K Pass@1 improves from 22.20 to 29.60 under wall-clock-matched continued pretraining (single run).
• DINOv3-ViT-L ADE20K mIoU improves from 51.33 to 52.47 and NYUv2 RMSE drops from 0.5752 to 0.5522 with ImageNet linear accuracy 84.07 to 84.59.
• Qwen1.5-4B GSM8K Pass@1 improves from 56.48 to 58.98; ablation shows random feedback (54.31) and uniform smoothing (54.58) fall below baseline.

Surprising findings

• Qwen1.5-0.5B drops on MMLU (38.08 to 35.01) under GSM8K feedback, indicating narrow steering harms small learners.
• Under the decontaminated NuminaMath-CoT stream the advantage shrinks (56.7 vs 57.5 for baseline vs V-pretraining), and the throughput cost (9.4%) is charged inside matched wall-clock budgets.

Limitations

• Local one-step value objective gives no long-horizon optimality guarantee; alignment can be noisy and depends on feedback quality.
• Adds computation overhead (9.4% throughput drop, 16.4% more peak memory) and may over-steer small learners toward narrow feedback tasks.
• Gains on some transfer benchmarks are mixed (MMLU drop for 0.5B, Oxford Hard slight decrease).

Experimental setup

Datasets: NuminaMath-CoT GSM8K MATH MMLU OMEGA ImageNet-1K ADE20K NYUv2 Oxford5k Paris6k OpenMathReasoning OpenWebMath

Benchmarks: GSM8K MATH MMLU OMEGA ADE20K NYUv2 ImageNet-1K Oxford5k Paris6k

Metrics: Pass@1 Pass@k mIoU RMSE linear accuracy mAP

Baselines: one-hot next-token continued pretraining DINOv3 vanilla view pipeline random feedback uniform top-K smoothing self-distillation SFT GRPO SFT+GRPO

Builds on: DINO DINOv3 I-JEPA influence-style bilevel approximation

Closest papers by meaning

• Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
• VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.87)
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.86)
• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.86)
• Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.86)
• Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.86)