Research map / Federated learning and privacy / Differential privacy
Learning from Aggregate responses: Instance Level versus Bag Level Loss Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows instance-level loss equals bag-level loss plus a data-dependent within-bag regularizer, proposes an interpolating estimator, and gives an asymptotic risk characterization for linear regression plus an optimal bag size analysis under label differential privacy.
Problem
Learning from aggregate responses (bags of samples with only averaged labels) under privacy constraints; the paper asks how bag-level and instance-level losses compare in generalization, and how to set bag size under label DP.
Approach
The instance-level loss equals the bag-level loss plus a feature-dependent within-bag variance penalty (for quadratic loss), enabling an interpolating estimator with parameter rho that trades bias for variance; precise high-dimensional risk is derived for linear models.
Key results
- Instance-level loss equals bag-level loss plus a within-bag regularizer R(theta) (exact for quadratic loss), with bag-level estimator unbiased and variance sigma^2/(psi/k - 1) versus instance-level variance sigma^2/(k(psi-1)).
- Theory matches simulations at d=100 for linear regression bias and variance under proportional asymptotics.
- On Boston Housing, optimal rho was 0 at bag size 80 and 1 at bag size 240, with intermediate optimal rho for sizes in between.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the interpolation parameter ρ from the bag-level loss toward the instance-level loss (ablation) | Interpolating estimator · Linear regression, proportional regime | bias | Hurt |
| Increase the interpolation parameter ρ from the bag-level loss toward the instance-level loss (ablation) | Interpolating estimator · Linear regression, proportional regime | variance | Helped |
| Use the instance-level loss instead of the bag-level loss (ablation) | Aggregate-response linear regression · Linear regression, proportional regime | variance | Helped |
| Use the instance-level loss instead of the bag-level loss (ablation) | Aggregate-response linear regression · Linear regression, proportional regime | prediction risk | Mixed |
| Use the bag-level loss rather than the instance-level loss when SNR is below the stated threshold (ablation) | Aggregate-response linear regression · Linear regression, ψ ≥ k > 1 | prediction risk | Helped |
| Use the instance-level loss rather than the bag-level loss when SNR is above the stated threshold (ablation) | Aggregate-response linear regression · Linear regression, ψ ≥ k > 1 | prediction risk | Helped |
| Increase bag size (ablation) | Interpolating estimator · Linear regression, proportional regime | bias | Hurt |
| Increase bag size at small ρ (ablation) | Interpolating estimator · Linear regression, proportional regime | variance | Hurt |
| Increase bag size at large ρ (ablation) | Interpolating estimator · Linear regression, proportional regime | variance | Helped |
| Increase the interpolation parameter ρ (ablation) | Interpolating estimator · Boston Housing | test-loss variance | Helped |
| Use stronger regularization as bag size increases (ablation) | Interpolating estimator · Boston Housing | test loss | Helped |
| Choose bag size k=1 when ρ is below the phase-transition threshold, and the largest candidate bag size when ρ is above it (ablation) | Differentially private interpolating estimator · Label-DP aggregate learning | prediction risk | Helped |
Surprising findings
- Under label DP, singleton bags are not always optimal; for large enough rho the largest bag size (k=5 in the studied range) minimizes risk.
Limitations
- Precise characterization holds only for linear models with Gaussian features in the proportional asymptotic regime.
- Instance-level bias result for general losses is only an inequality; the exact decomposition is restricted to quadratic loss.
Experimental setup
Datasets: Boston Housing synthetic Gaussian linear regression data
Benchmarks: Boston Housing
Metrics: prediction risk test loss bias variance
Baselines: bag-level loss estimator instance-level loss estimator
Builds on: learning from label proportions empirical proportion risk minimization label differential privacy convex gaussian min-max theorem
Closest papers by meaning
- PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses (ICML 2024 · similarity 0.92)
- Differentially Private Bagging: Improved utility and cheaper privacy than subsample-and-aggregate (NeurIPS 2019 · similarity 0.86)
- It's My Data Too: Private ML for Datasets with Multi-User Training Examples (ICML 2025 · similarity 0.84)
- Mitigating Privacy Risk in Membership Inference by Convex-Concave Loss (ICML 2024 · similarity 0.83)
- Assessing Per-Sample Membership Inference Vulnerability without Retraining (NeurIPS 2026 · similarity 0.83)
- Leveraging Per-Instance Privacy for Machine Unlearning (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.