Research map / Label-efficient and robust learning / Knowledge distillation

Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Few-shot and metric distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives an exact total-variation identified-set diameter for top-K censored LLM logit APIs, gives a binary-endpoint KL lower bound and small-ambiguity upper bound, and shows on Qwen3-0.6B math/code that per-position distribution recovery decouples from capability transfer.

Problem

Top-K censored LLM APIs reveal only the K highest logits; the question is how much per-position distribution (and downstream capability) an attacker can recover from that access model.

Approach

Treats top-K censoring as partial identification: the set of teacher distributions consistent with the observed top-K logits has an exact TV diameter U_K computable from one API call, plus a KL minimax lower bound via a binary-endpoint argument, and separates fidelity (KL) from capability transfer (PVR).

Key results

• Exact TV diameter of identified set at K=20 averages U_K=0.908 on Qwen3-0.6B math teacher; even at K=100 mean U_K=0.809.
• On GSM8K, on-task top-K KD recovers 11.8% PVR, on-task full-logit KD 56.5%, and generation SFT 96.1%, while on-task full-logit KD reaches 99.0% KL closure.
• Off-task sparse strict_topk_kd reaches 27.8% PVR despite -14% KL closure; delta_rank gets 60.0% KL closure but only 7.2% PVR.

Surprising findings

• Sparse top-K KD achieves 27.8% PVR despite negative KL closure, while the best-KL delta_rank variant gets only 7.2% PVR.
• On the Qwen3-1.7B to 0.6B stress test, top-K methods do not beat CE-only, while full-logit DKD still gives a small KL reduction.

Limitations

• Bounds are per-position; adaptive composition, autoregressive logit access, or generation can tighten or bypass the identified set.
• Finite-U_K KL minimax rate is not established; the binary lower bound is conservative at observed U_K and the upper bound is only asymptotic.
• PVR validated mainly within one model family and scale; API rounding, quantization, or temperature would change the geometry.

Experimental setup

Datasets: GSM8K MBPP WikiText-103

Benchmarks: GSM8K MBPP

Metrics: TV diameter U_K KL divergence KL closure private value recovery (PVR) exact match accuracy execution-pass rate top-1 agreement

Baselines: ce_only strict_topk_kd bild delta_rank dkd_full ontask_topk ontask_dkd gen_sft joint_gen_kd

Builds on: knowledge distillation generalized knowledge distillation (GKD) DeepSeek-R1 distillation

Closest papers by meaning

• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.83)
• Instance-Optimality for Private KL Distribution Estimation (NeurIPS 2025 · similarity 0.83)
• Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss (ICML 2025 · similarity 0.83)
• Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.83)
• Distillation Robustifies Unlearning (NeurIPS 2025 · similarity 0.83)
• The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.83)