Research map / Large language models / Language, safety and interpretability

ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentences

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Text generation and evaluation metrics

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ImpScore, a reference-free learnable metric that scores sentence implicitness as the divergence between pragmatic and semantic embeddings, trained with pairwise contrastive learning on 112,580 implicit-explicit pairs, and applies it to hate speech datasets and LLM evaluation.

Problem

Measuring the implicitness level of sentences on a continuum without reference texts; existing benchmarks rely on binary, subjective implicit/explicit labels, and there is no automatic metric for sentence-level implicitness.

Approach

Define implicitness as the divergence between semantic meaning and pragmatic interpretation; project Sentence-BERT embeddings into separate pragmatic and semantic spaces, map pragmatics into the semantic space, and score implicitness as cosine distance, trained with margin-based pairwise ranking losses plus a pragmatic-distance loss.

Key results

• ImpScore achieves over 95% accuracy on both implicitness and pragmatics accuracy on the held-out test set.
• In the OOD user study, ImpScore's Ranking Kendall's Tau/Spearman's Rho are comparable to individual human participants (e.g., 0.73/0.80 on Set 1 and 0.80/0.88 on Set 2), and it matches humans at 0.80 and 1.00 on Choice Set 1 and Set 2.
• LLM hate speech detection accuracy declines as implicitness increases on ToxiGen and DGHSD for GPT-4-Turbo, Llama-3.1-8B-Instruct, and OpenAI Moderation.

Measured effects reported in the paper

Surprising findings

• In the Adversarial dataset, GPT-4-Turbo and Llama-3.1-8B-Instruct improve on the highest implicitness range [1.75, 2], attributed to low sample diversity there.
• ToxiGen, generated by prompting GPT for implicit hate speech, shows low implicitness scores, suggesting LLMs struggle to generate human-level implicit language.

Limitations

• Training dataset is relatively small (112,580 pairs) and relies on GPT-generated explicit sentences, limiting generalizability.
• User study is small (10 English-speaking participants, 20 topics) and English-only.
• The metric sometimes misranks sentences with minimal implicitness differences.

Experimental setup

Datasets: ToxiGen Latent Hatred DGHSD Adversarial Implicit Hate Speech SBIC Implicatures ImpPres LUDWIG ABSA SemEval-2018 Task 3 Snarks R^3

Benchmarks: ImpScore held-out test set OOD user study (20 topics, Ranking and Choice tasks) ToxiGen DGHSD Adversarial Implicit Hate Speech

Metrics: Implicitness Accuracy Pragmatics Accuracy Kendall's Tau Spearman's Rho Accuracy

Baselines: GPT-4-Turbo Llama-3.1-8B-Instruct OpenAI Moderation

Builds on: Sentence-BERT GPT-3.5 Xavier initialization

Closest papers by meaning

• Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution (AAAI 2024 · similarity 0.86)
• Paloma: A Benchmark for Evaluating Language Model Fit (NeurIPS 2024 · similarity 0.84)
• DiffScore: Text Evaluation Beyond Autoregressive Likelihood (NeurIPS 2026 · similarity 0.84)
• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.84)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.84)
• Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.84)