Research map / Large language models / Language, safety and interpretability
SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SNS-Bench, a 6,658-question benchmark over 8 social networking service tasks, and evaluates 25+ open and closed LLMs on it.
Problem
LLM evaluation for social networking services (SNS) has focused on isolated tasks and lacks a comprehensive, standardized benchmark covering note comprehension, retrieval, sentiment/intent, and recommendation.
Approach
Defines LLM capabilities for SNS as four groups (note comprehension, information retrieval, sentiment and intent analysis, personalized recommendation), operationalizes them into 8 tasks built from platform data with a five-step curation pipeline, and scores each task with tailored metrics (accuracy, F1, macro-F1, SR-weighted scores, ANLS plus semantic similarity).
Key results
- Claude-3.5-Sonnet leads with average score 61.98%, closely followed by GPT-4o-2024-05-13 (61.50%) and the best open model Qwen-2.5-72B-Instruct at 61.27%.
- Closed-source models outperform open-source ones, with the top-open vs top-closed gap about 1%.
- Note-MRC (Complex) and Note-QueryGen are the hardest tasks, while Note-Taxonomy (Single) and Note-Hashtag (Single) are easier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use larger Qwen2.5-Instruct models | Qwen-2.5-Instruct · SNS-BENCH average | average score: 52.86 -> 58.89 -> 61.27 | Helped |
| Use a closed-source model rather than an open-source model | LLMs evaluated on SNS-BENCH · SNS-BENCH | average score | Helped |
| Use larger model architectures | Open-source LLMs evaluated on SNS-BENCH · SNS-BENCH | average score: 61.27 | Helped |
Surprising findings
- Closed-source leader and top open-source model differ by only about 1% in average score.
- GLM-4 struggles notably with list-format output instruction following.
Limitations
- Primarily text-based, overlooking multimodal aspects of SNS content.
- Predominantly English-based dataset, though the data comes from a Chinese platform, limiting geographic and linguistic coverage.
- Metrics may not fully capture cultural and contextual nuances.
Experimental setup
Datasets: SNS-Bench (6,658 questions from a social platform)
Benchmarks: SNS-Bench
Metrics: accuracy F1 macro-F1 ANLS semantic similarity ROUGE BLEU success ratio
Baselines: GPT-4o-2024-11-20 Claude-3.5-Sonnet Gemini-1.5-Pro Deepseek-V3 Qwen-2.5-72B-Instruct Llama-3.3-70B-Instruct
Builds on: OpenCompass
Closest papers by meaning
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.87)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.86)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.85)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.85)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.85)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.