Research map / Label-efficient and robust learning / Recognition and long-tail
WRENCH: A Comprehensive Benchmark for Weak Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| no single weak supervision method wins across datasets | WS methods · classification benchmarks | best method per dataset | Mixed |
| fine-tune a pre-trained language model as end model (vs label model alone) | weak supervision on text · text classification datasets | accuracy/F1 | Helped |
| use COSINE on top of BERT/RoBERTa end model | fine-tuned LM end model · text classification | accuracy/F1 | Helped |
| use soft labels instead of hard labels for deeper end models (ablation) | end model training · classification averaged across datasets | average performance | Helped |
| use CHMM (data-feature-aware HMM) instead of token-independent label models | sequence tagging label model · 8 NER datasets average | entity F1: 64.38 -> 66.32 | Helped |
| train neural end model on weak labels instead of applying label model directly | sequence tagging with LSTM-CNN · 8 NER datasets average | entity F1: 66.32 -> 62.97 | Hurt |
| use joint model ConNet instead of two-stage with CHMM | BERT end model · 8 NER datasets average | entity F1: 66.50 -> 63.81 | Hurt |
| weak labels vs gold labels for BERT end model | BERT sequence tagger · 8 NER datasets average | entity F1: 84.36 -> 66.50 | Hurt |
| increase variance of LF accuracy (radius) in synthetic LFs (ablation) | label models · synthetic binary LFs | AUC | Mixed |
| increase LF propensity in synthetic LFs (ablation) | label models · synthetic binary LFs | AUC | Helped |
Closest papers by meaning
- Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks (NeurIPS 2024 · similarity 0.93)
- Characterizing the Impacts of Semi-supervised Learning for Weak Supervision (NeurIPS 2023 · similarity 0.92)
- AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels (NeurIPS 2022 · similarity 0.92)
- Weak Supervision Performance Evaluation via Partial Identification (NeurIPS 2024 · similarity 0.88)
- Universalizing Weak Supervision (ICLR 2022 · similarity 0.88)
- CGMatch: A Different Perspective of Semi-supervised Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.