lesakbota/kazakhstan-sociology-llm-benchmark

Kazakhstan Sociology Consultant — LLM Benchmark Dataset

Benchmark dataset for evaluating Large Language Models on sociological survey data analysis tasks (Kazakhstan).

Diploma thesis: "Implementation of a visual-statistical analytics module in a digital sociology consultant system"

Dataset Description

This benchmark evaluates LLMs on their ability to:

• —Parse natural language queries (Russian) about sociological data
• —Generate correct SQLite SQL queries with JOINs
• —Choose appropriate chart types (bar / pie / line)
• —Return structured JSON responses

Test Cases: 30 total

Authorship

• —Hand-crafted: 30/30 test cases (queries, expected types, SQL keywords)
• —Auto-generated: Model responses (SQL, chart_type, explanation) — produced by LLMs during evaluation

Models Evaluated

Metrics (7 binary metrics per test case)

Score = average of applicable metrics × 100%

Files

Database Schema

The benchmark uses a real Kazakhstan sociological survey database:

• —3,000 respondents, 206 variables, 618,000 responses
• —Tables: datasets, variables, value_labels, respondents, response_values
• —Note: The database itself is NOT included (private survey data). Only the benchmark queries and results are published.

Usage

import pandas as pd

# Load test cases
cases = pd.read_csv("test_cases.csv")

# Load model results
gemma = pd.read_csv("gemma3n-latest.csv")
gpt = pd.read_csv("gpt-4o-mini.csv")

# Compare scores
print(gemma[["id", "query", "score"]].head())

Citation

If you use this benchmark, please cite:

@misc{sociology_llm_benchmark_2026,
  title={LLM Benchmark for Kazakhstan Sociological Survey Analysis},
  year={2026},
  publisher={Hugging Face},
}

Source Code

Full system source code: lesakbota/kazakhstan-sociology-consultant

License

MIT License