LivXue/Social-Media-Agents-Benchmark

🤖 SoMe: A Realistic Benchmark for LLM-based Social Media Agents

<div align="center" style="line-height: 1.5;">

  

</div>

📋 Overview

SoMe is a comprehensive benchmark designed to evaluate the capabilities of Large Language Model (LLM)-based agents in realistic social media scenarios. This benchmark provides a standardized framework for testing and comparing social media agents across multiple dimensions of performance.

SoMe comprises a diverse collection of:

• —8 social media agent tasks
• —9,164,284 posts from various social media platforms
• —6,591 user profiles with rich behavioral data
• —25,686 reports from external websites
• —17,869 meticulously annotated task queries

📰 News

• —[2025.11] 🎉 Our paper is accepted by AAAI 2026!

✨ Features

SoMe benchmark evaluates social media agents across 8 key tasks, covering diverse aspects of social media intelligence:

📈 Dataset Statistics

The SoMe benchmark includes comprehensive datasets for each task, with the following statistics:

📁 Project Structure

Social-Media-Agent/
├── 🤖 agent.py                    # Main social media agent implementation
├── 🔧 qwen_agent/                 # Qwen-Agent library
├── 📋 tasks/                      # Task-specific modules
│   ├── 📱 media_content_recommend/
│   ├── 🚫 misinformation_detection/
│   ├── 🚨 realtime_event_detection/
│   ├── ❓ social_media_question_answering/
│   ├── 📊 streaming_event_summary/
│   ├── 💬 user_comment_simulation/
│   ├── 😊 user_emotion_analysis/
│   └── 🎯 user_behavior_prediction/
├── 🛠️ tools/                      # Tools for social media analysis
├── 🧪 test_*.py                   # Test scripts for each task
├── 📊 eval_scripts/               # Evaluation scripts for scoring
├── 📂 results/                    # Directory for storing results
├── 📊 datasets/                   # Dataset directory
└── 💾 database/                   # Database directory

🚀 Installation

Prerequisites

• —Python 3.12+ installed on your system
• —Git installed for repository cloning
• —Sufficient disk space for data (recommended: 50GB+)

Installation Steps

1. 1.📥 Clone the repository

   git clone https://github.com/LivXue/SoMe.git
   cd SoMe

1. 1.📦 Install dependencies

   pip install -r requirements.txt

1. 1.📥 Download test data
2. 2.Hugging Face Dataset: Download Link
3. 3.Google Drive: Download Link
4. 4.Baidu Disk: Download Link (Password: SoMe)

After downloading, unzip the data into the database directory.

💻 Usage

🏃‍♂️ Running Individual Tasks

Each task can be evaluated using its corresponding test script:

# 🚨 Realtime Event Detection
python test_realtime_event_detection.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 📊 Streaming Event Summary
python test_streaming_event_summary.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 🚫 Misinformation Detection
python test_misinformation_detection.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 🎯 User Behavior Prediction
python test_user_behavior_prediction.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 😊 User Emotion Analysis
python test_user_emotion_analysis.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 💬 User Comment Simulation
python test_user_comment_simulation.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# 📱 Media Content Recommendation
python test_media_content_recommend.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

# ❓ Social Media Question Answering
python test_social_media_question_answering.py --model MODEL_NAME --base_url MODEL_SERVER_URL --api_key API_KEY

⚙️ Command Line Arguments

📊 Evaluation

After running the test scripts, evaluate the results using the provided evaluation scripts:

# Option 1: For tasks with LLM-based answer extraction
python eval_scripts/[TASK]_extraction.py
python eval_scripts/[TASK]_compute_score.py

# Option 2: For tasks with LLM-as-judge scoring
python eval_scripts/[TASK]_scoring.py
python eval_scripts/[TASK]_compute_score.py

Note: The LLM settings for evaluation are configured in eval_scripts/settings.json

🧠 Model Support

The benchmark supports various LLM models through OpenAI-compatible API endpoints:

• —🧩 Qwen series models (Qwen2.5, Qwen3, etc.)
• —🔌 OpenAI models (GPT-4, GPT-5, etc.)
• —🌐 Third-party models with OpenAI-compatible APIs (DeepSeek, Claude, etc.)
• —📦 Local models served with OpenAI-compatible wrappers (vLLM, Ollama, etc.)

📚 Citation

If you use this benchmark in your research, please cite our paper:

@inproceedings{some2026,
  title={SoMe: A Realistic Benchmark for LLM-based Social Media Agents},
  author={Dizhan Xue and Jing Cui and Shengsheng Qian and Chuanrui Hu and Changsheng Xu},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2026}
}

🤝 Contributing

We welcome contributions to improve the benchmark! Here's how you can help:

1. 1.🐛 Report bugs by opening issues with detailed descriptions
2. 2.💡 Suggest features for new tasks or improvements
3. 3.🔧 Submit code via pull requests for bug fixes or enhancements
4. 4.📊 Add datasets to expand the benchmark coverage
5. 5.📝 Improve documentation for better usability

Please see our Contributing Guidelines for more details.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🙏 Acknowledgments

We would like to express our gratitude to:

• —The Qwen team for their excellent Qwen-Agent framework, which forms the foundation of this benchmark
• —All contributors who have helped develop and improve SoMe
• —The social media platforms and data providers that make this research possible
• —The AAAI 2026 reviewers for their valuable feedback

📞 Contact

For questions or inquiries about the benchmark, please contact:

• —Dizhan Xue: xuedizhan17@mails.ucas.ac.cn

Visit our GitHub repository for the latest updates and discussions.