AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning

Jaehyung Kim, Jinwoo Shin, Dongyeop Kang

ICML 2023 · Label-efficient and robust learning › Recognition and long-tail · approach: Loss design and classifier theory · 0 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add auxiliary preference learning (P2C) with GPT-3 generative preference labelsRoBERTa-base classifier · CoLA, SMS Spam, Hate Speech, Emotiontest accuracy / Mcc / ECE: e.g., Mcc 63.7 to 65.4; ECE 3.6 to 2.8 on CoLA; ~11.55% relative test error reduction on averageImproved
add P2C with extractive preference from annotation recordsRoBERTa-base classifier · Offensive, Polite-Wiki, Polite-SE, MNLI, DynaSent-R1, DynaSent-R2test accuracy: improves on all six datasets; ~7.59% relative test error reduction vs vanilla on averageImproved
P2C with extractive preference vs. disagreement learning baselines (soft-labeling, margin, filtering, weighting, multi-annotator, CS-KD)RoBERTa-base classifier · six annotation-record datasetstest accuracy: 4.27% relative test error reduction vs previous best disagreement method on averageImproved
add subjective (crowd-sourced) preference labels vs. 5k extra task labels vs. generative/extractive preferenceRoBERTa-base on DynaSent-R2 · DynaSent-R2test accuracy and ECE: subjective 71.74 / ECE 6.09 vs task labels 71.17 / ECE 9.19Improved
add 5k GPT-3 generative preference labels instead of 5k additional task labelsRoBERTa-base on DynaSent-R2 · DynaSent-R2test accuracy: generative 71.46 vs task labels 71.17Improved
use multiple (T=3) preference heads instead of a single head (ablation)P2C without div/cons/sampling · DynaSent-R2test accuracy: 71.84 (T=1) to 71.92 (T=3)Improved
add diversity regularization L_div between preference heads (ablation)P2C · DynaSent-R2test accuracy: 71.92 to 72.05Improved
add diversity regularization L_div between preference heads (ablation)P2C · Offensivetest accuracy: 76.43 to 76.67Improved
add consistency regularization L_cons between classification and preference heads (ablation)P2C · DynaSent-R2test accuracy: 72.05 to 72.67Improved
add consistency regularization L_cons between classification and preference heads (ablation)P2C · Offensivetest accuracy: 76.67 to 77.67Improved
add disagreement/inconsistency-based informative pair sampling (ablation)P2C · DynaSent-R2 and Offensivetest accuracy: DynaSent-R2 72.67 to 73.06; Offensive 77.67 to 77.81Improved
add P2C with extractive preference to ResNet-18 scene attribute classificationResNet-18 on SUN Attribute · five binary attribute taskstest accuracy: 76.48 to 78.05 average (~6.66% relative test error reduction)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy