jacobli/studybench

1---2license: cc-by-4.03language:4- en5tags:6- code7- question-answering8- llm-evaluation9- rubric-grading10- agents11task_categories:12- question-answering13- text-generation14size_categories:15- n<1K16configs:17- config_name: dspy18  data_files: dspy.jsonl19- config_name: openclaw20  data_files: openclaw.jsonl21---22 23# studybench24 25**studybench** is a small, high-effort benchmark of **expert-level coding questions** about real26open-source codebases, each paired with a **gold answer** and a **weighted, source-grounded grading27rubric**. The questions ask a model to produce working code that uses a specific library/framework28correctly; the rubric decomposes a correct answer into discrete, checkable claims, each tied to exact29lines of the upstream source.30 31This release publishes **both the questions and the full rubrics** (nothing is held back), so the32evaluation is fully transparent and reproducible.33 34## Configs35 36Pick a subset with the second argument of `load_dataset`:37 38```python39from datasets import load_dataset40 41dspy     = load_dataset("jacobli/studybench", "dspy")        # 30 questions42openclaw = load_dataset("jacobli/studybench", "openclaw")    # 20 questions43```44 45| config | questions | topics | codebase |46|---|---:|---:|---|47| `dspy` | 30 | 6 | [DSPy](https://github.com/stanfordnlp/dspy) |48| `openclaw` | 20 | 4 | [OpenClaw](https://github.com/openclaw/openclaw) |49 50## Schema51 52Each row has six fields:53 54| field | type | description |55|---|---|---|56| `id` | string | stable opaque identifier |57| `topic` | string | coarse category (see below) |58| `question` | string | the task prompt — asks for a self-contained, runnable solution |59| `gold_answer` | string | a reference solution (code) |60| `rubric` | list | weighted claims that define a correct answer |61| `evidence` | list | source excerpts that ground the rubric |62 63**`rubric`** — a list of claims; weights sum to **100** per question:64 65```json66{67  "claim_id": "c1",68  "claim_type": "core",          // "core" = essential; "supporting" = secondary69  "weight": 52,                   // integer; the rubric's weights sum to 10070  "statement": "…what must be true of a correct answer…",71  "span_ids": ["s4", "s8"]        // evidence spans grounding this claim72}73```74 75**`evidence`** — the source excerpts the grader is shown; every `span_ids` value in `rubric`76resolves to one of these `span_id`s:77 78```json79{80  "span_id": "s4",81  "path": "dspy/teleprompt/gepa/gepa.py",   // path within the upstream repo82  "start_line": 330,83  "end_line": 365,84  "excerpt": "0330:     def __init__(\n0331:         self,\n…"   // line-number-prefixed source85}86```87 88Excerpts are byte-exact copies of the upstream source at the pinned commits below (each line is89prefixed with its 1-indexed line number, e.g. `0330: `).90 91### Topics92- **dspy:** `gepa_optimizer_usage`, `prompt_optimization_workflows`, `rag_and_retrieval_pipelines`, `react_agents_and_tools`, `signature_schema_and_pydantic_types`, `evaluation_metrics_and_custom_eval`93- **openclaw:** `model_fallback_and_failover_logic`, `cross_session_channel_context_and_session_behavior_requests`, `memory_core_dreaming_and_promotion_pipeline`, `new_plugin_provider_and_channel_integration_requests`94 95## How the rubric is used for grading96 97A judge model is shown the **question**, the **candidate answer**, the **`gold_answer`**, the98**`rubric`**, and the **`evidence`** spans. It scores each claim independently (does the answer99satisfy the claim?), and the question score is the **weight-weighted fraction of satisfied claims**100(0–100). `claim_type` lets you apply an optional **conjunctive gate**: require every `core` claim to101be satisfied or the answer scores 0. The `evidence` excerpts are the *only* code context the judge102needs — grading does not require checking out the repositories.103 104## Source code & attribution105 106The `evidence` excerpts and `path` values reference these repositories at fixed commits:107 108| codebase | repo | commit | license |109|---|---|---|---|110| DSPy | `stanfordnlp/dspy` | `9cdb0aac28b2a04b064e40697ccd301872cf6a43` | MIT |111| OpenClaw | `openclaw/openclaw` | `da228660306b55a9cce3b973946f3aacfc515848` | MIT |112 113To inspect or extend the evidence, check out the corresponding repo at the pinned commit and open114the listed `path` at the given line range.115 116## Licensing117 118- **Questions, gold answers, and rubrics** (the original contributions of this dataset) are released119  under **CC-BY-4.0**.120- **Embedded source `excerpt`s** are derived from DSPy and OpenClaw and remain under their respective121  **MIT** licenses; attribution is provided above.122 123## Notes & limitations124 125- This is a deliberately small, expert-curated set (50 questions total), not a large-scale benchmark.126- Because both questions and rubrics are public, treat results as an **open** (non-held-out)127  evaluation; models may be trained on this content.128- The benchmark is grounded in specific repository snapshots; answers reflect the APIs at the pinned129  commits.130