Research map / Large language models / Agents, code and math
SoMe: A Realistic Benchmark for LLM-based Social Media Agents
Dizhan Xue, Jing Cui, Shengsheng Qian, Chuanrui Hu, Changsheng Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale model size from Qwen3-14B to Qwen3-32B | Qwen3 agent · SoMe average score | average score: 52.82 -> 53.98 | Improved |
| scale model size from Qwen3-8B to Qwen3-14B | Qwen3 agent · SoMe average score | average score: 50.89 -> 52.82 | Improved |
| scale from Llama-3.1-8B to Llama-3.3-70B | Llama agent · SoMe average score | average score: 31.65 -> 48.34 | Improved |
| use reasoning-distilled DeepSeek-R1-0528-Qwen3-8B instead of Qwen3-8B | 8B agent · SoMe average score | average score: 50.89 -> 31.87 | Hurt |
| use larger GLM-4-32B instead of GLM-4-9B | GLM-4 agent · SoMe average score | average score: 37.51 -> 38.86 | Improved |
| larger model size reduces tool-response hallucination | Kimi-K2-Instruct (1T) and DeepSeek-V3 (671B) · SoMe Real-time Event Detection hallucination test | tool response hallucination rate: 7% and 13% | No clear effect |
| higher task completion rate (tool-calling ability) goes with better scores | Gemini-2.5-Flash, DeepSeek-V3, Qwen3-32B, Qwen3-14B · SoMe | average score: TCR 96.61 to 98.60 | Improved |
Closest papers by meaning
- SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services (ICML 2025 · similarity 0.86)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.86)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.86)
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.85)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.85)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
