Research map / Label-efficient and robust learning / Recognition and long-tail
Weak Supervision Performance Evaluation via Partial Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames weak-supervision model evaluation as partial identification and estimates Fréchet bounds on accuracy, precision, recall and F1 via smoothed convex dual programs, without ground-truth labels.
Problem
Performance metrics (accuracy, precision, recall, F1) of classifiers trained with programmatic weak supervision cannot be identified without ground-truth labels, since only the marginals P(X,Z) and P(Y,Z) are known.
Approach
Estimate Fréchet bounds on E[g(X,Y,Z)] over all joints consistent with the known marginals by reformulating them as smooth (softmin/softmax) finite-dimensional convex dual programs, solved with L-BFGS, with asymptotic normality results for confidence intervals.
Key results
- On AGNews, Snorkel-based bounds for accuracy were 0.42 (lower) to 0.90 (upper) versus oracle-based 0.46 to 0.95, with test accuracy 0.76 inside the interval.
- Model selection by Fréchet lower bound or bounds average matched or beat a label-model baseline and a labeled validation set (n=100) on several datasets, e.g., commercial F1 0.96 vs 0.91 for labeled set.
- On YouTube with few-shot LLM weak labels, adding high-quality WRENCH labels significantly shrank and shifted the bounds upward, consistent with test accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select an MLP using the Fréchet lower bound instead of a labeled validation set of size n=100 | MLP model selection · Tennis | F1: $0.71_{\pm 0.02}$ -> $0.76_{\pm 0.01}$ | Helped |
| Select an MLP using the Fréchet lower bound instead of a labeled validation set of size n=100 | MLP model selection · Commercial | F1: $0.91_{\pm 0.01}$ -> $0.96_{\pm 0.00}$ | Helped |
| Select an MLP using the Fréchet lower bound rather than label-model estimates | MLP model selection · Tennis | F1: $0.75_{\pm 0.01}$ -> $0.76_{\pm 0.01}$ | Helped |
| Add high-quality weak labels from Wrench to the few-shot LLM weak labels | YouTube classifier using Llama-2-13b-chat-hf weak labels · YouTube | accuracy | Helped |
| Add synthetic low-quality weak labels to the few-shot LLM weak labels | YouTube classifier using Llama-2-13b-chat-hf weak labels · YouTube | accuracy | No clear effect |
| Estimate P(Y|Z) with Snorkel rather than true labels | Fréchet performance bounds · Wrench datasets | accuracy bounds | No clear effect |
Surprising findings
- Snorkel's default label model, despite possible misspecification, yields bounds comparable to those from an oracle label model estimated with true labels.
Limitations
- Applies only when the outcome space Y and weak label space Z are finite.
- Bounds can be wide when |Z| is large or weak labels are uninformative; convergence may be slow.
- Asymptotic guarantees assume label model consistency; misspecification shifts bounds by O(delta).
Experimental setup
Datasets: WRENCH benchmark (Census, YouTube, SMS, IMDB, Yelp, AGNews, TREC, Spouse, SemEval, CDR, ChemProt, Commercial, Tennis Ral Hate Speech dataset
Benchmarks: WRENCH datasets Hate Speech dataset
Metrics: accuracy precision recall F1 score coverage of bounds
Baselines: Snorkel labeled validation set
Builds on: Snorkel Fréchet bounds noise-aware loss
Closest papers by meaning
- Characterizing the Impacts of Semi-supervised Learning for Weak Supervision (NeurIPS 2023 · similarity 0.90)
- AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels (NeurIPS 2022 · similarity 0.90)
- Universalizing Weak Supervision (ICLR 2022 · similarity 0.89)
- Learning from weak labelers as constraints (ICLR 2025 · similarity 0.89)
- Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods (ICML 2020 · similarity 0.88)
- A General Framework for Learning from Weak Supervision (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.