Research map / Large language models / Retrieval and knowledge
SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SubjECTive-QA, a manually annotated dataset of earnings call QA answers rated on six subjectivity features, and benchmarks PLMs and LLMs on it, with transfer tests on White House press briefings.
Problem
Soft misinformation in formal QA settings (e.g., earnings calls) is factually correct but may lack clarity, relevance, or specificity, and there are no manually annotated datasets for subjective features of such answers across multiple dimensions.
Approach
A six-feature subjectivity annotation scheme (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) applied to long-form earnings call QA pairs, with features chosen via PaLM-2-suggested properties filtered by Loughran-McDonald word lists and expert selection, and benchmarked across fine-tuned PLMs and LLMs.
Key results
- RoBERTa-base had the highest average weighted F1 of 63.95% across six features, while Llama-3-70b-Chat reached 82.75% weighted F1 on Relevant and BERT reached 80.93% on Clear.
- Transfer test of RoBERTa-base on 65 QA pairs from White House Press Briefings and Gaggles gave a mean weighted F1 of 65.97%.
- Models performed better on low-subjectivity features (Relevant, Clear) than on Specific and Assertive (mean difference of 10.01% in weighted F1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BERT base (uncased) for Clear classification | SubjECTive-QA models · SubjECTive-QA — Clear | weighted F1: 80.93\% | Helped |
| Use RoBERTa-base for Optimistic classification | SubjECTive-QA models · SubjECTive-QA — Optimistic | weighted F1: 62.69\% | Helped |
| Use RoBERTa-base for Assertive classification | SubjECTive-QA models · SubjECTive-QA — Assertive | weighted F1: 49.10\% | Helped |
| Use Llama-3-70b-Chat for Relevant classification | SubjECTive-QA models · SubjECTive-QA — Relevant | weighted F1: 82.75\% | Helped |
| Use BERT base (uncased) for Cautious classification | SubjECTive-QA models · SubjECTive-QA — Cautious | weighted F1: 60.66\% | Helped |
| Use RoBERTa-base across features | SubjECTive-QA models · SubjECTive-QA — all six features | average weighted F1: 63.95\% | Helped |
| Fine-tune RoBERTa-base on SubjECTive-QA before testing on political QA | RoBERTa-base · White House Press Briefings and Gaggles | mean weighted F1: 65.97\% | Helped |
Surprising findings
- Fine-tuned general-purpose PLMs such as RoBERTa-base matched or beat much larger LLMs on several features, including overall average.
- Mixtral-8x22B had a higher average weighted F1 than Llama-3-70b-Chat despite the latter being a stronger model in other respects.
Limitations
- Covers only 120 NYSE-listed earnings calls from 2007 to 2021 and US companies, limiting generalizability.
- Annotations were produced by nine authors who share similar educational and demographic backgrounds, introducing bias.
- Uses written transcripts, so audio cues such as tone and intonation are not captured.
Experimental setup
Datasets: SubjECTive-QA White House Press Briefings Gaggles
Benchmarks: SubjECTive-QA
Metrics: weighted F1
Baselines: BERT base (uncased) FinBERT-tone Llama-3-70b-Chat Llama-3-8b-Chat Mixtral-8x22B Instruct Mixtral-8x7B Instruct GPT-4o
Builds on: Loughran and McDonald Sentiment Word Lists FinBERT
Closest papers by meaning
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.82)
- B-score: Detecting biases in large language models using response history (ICML 2025 · similarity 0.82)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.81)
- The SEC Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data (NeurIPS 2026 · similarity 0.81)
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.81)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.