Research map / Large language models / Language, safety and interpretability
BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BBScore, a reference-free text coherence metric based on a modified Brownian bridge over frozen GPT-2 contrastive latents, optionally combined with a small MLP classifier, evaluated on shuffle, local-window, and LLM-vs-human discrimination tasks.
Problem
Measuring global and local coherence of long texts; existing neural coherence models rely on static embeddings or narrow local context, and end-to-end pairwise-trained models generalize poorly to real incoherent text.
Approach
Model sentence latents from a frozen GPT-2 encoder trained with a Brownian-bridge contrastive objective as a Brownian bridge trajectory; estimate its diffusion coefficient by maximum likelihood on training latents and score documents by the resulting log-likelihood deviation (BBScore), with an optional three-layer perceptron on multi-window BBScores.
Key results
- On the global block-shuffle test, BBScore + CLF reaches about 98.5% test accuracy across block sizes 1, 2, 5, 10, comparable to Unified Coherence (99.73 on block 1, 96.09 on block 10).
- Raw BBScore (no classifier) achieves AUC 0.835 on block-1 shuffle and 0.738 on block-10 shuffle.
- On AI generated text differentiation, BBScore + CLF reaches 83-92% pairwise accuracy across six LLMs, versus 18-55% for Entity Grid on train split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a classifier to BBScore | BBScore · WikiSection global discrimination test D_b=10 | accuracy: 78.66 -> 98.50 | Helped |
| Add a classifier to BBScore | BBScore · WikiSection local discrimination test D_w=1,2,3 | accuracy: 57.66 -> 67.12 | Helped |
| Add a classifier to BBScore | BBScore · WikiSection AI differentiation test LLaMA2-7B | accuracy: 75.63 -> 92.02 | Helped |
| Use BBScore + CLF instead of UnifiedCoherence | WikiSection global discrimination · WikiSection global discrimination test D_b=10 | accuracy: 96.09 -> 98.50 | Helped |
| Use BBScore + CLF instead of UnifiedCoherence | WikiSection local discrimination · WikiSection local discrimination test D_w=1,2,3 | accuracy: 87.00 -> 67.12 | Hurt |
| Add a classifier to BBScore | BBScore · WikiSection AI differentiation test GPT-NeoX | accuracy: 83.74 -> 86.17 | Helped |
| Use the approximated diffusion coefficient instead of the standard Brownian bridge coefficient σ²_m=1 | BBScore · WikiSection block 1 shuffle test | AUC | Helped |
Surprising findings
- The standard Brownian bridge with sigma_m^2 = 1 gives AUC of 1 on block-1 shuffle in the appendix analysis, showing that the diffusion coefficient estimation is essential.
- On GCDC three-class classification with the WikiSection encoder, accuracies are near chance (about 41-51%).
Limitations
- Evaluated on a single domain (WikiSection city) and the assumption that sigma_m^2 encodes domain/style is restrictive.
- Out-of-domain generalization to GCDC is poor, and LLM detection results are not entirely satisfactory.
- No correlation with formal human evaluation of coherence has been examined.
Experimental setup
Datasets: WikiSection (city) GCDC
Benchmarks: WikiSection shuffle (global discrimination) WikiSection local window shuffle (local discrimination) WikiSection AI-generated text differentiation GCDC (domain generalizability)
Metrics: pairwise accuracy AUC classification accuracy Wasserstein distance
Baselines: Entity Grid Unified Coherence BBScore (no classifier)
Builds on: Unified Coherence model Brownian bridge encoder contrastive learning (prior long-text generation work) Entity Grid
Closest papers by meaning
- Correlation Dimension of Autoregressive Large Language Models (NeurIPS 2025 · similarity 0.85)
- DiffScore: Text Evaluation Beyond Autoregressive Likelihood (NeurIPS 2026 · similarity 0.85)
- Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models (ICML 2024 · similarity 0.84)
- Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education (AAAI 2024 · similarity 0.84)
- VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models (ICLR 2025 · similarity 0.84)
- Open-Domain Text Evaluation via Contrastive Distribution Methods (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.