libredb/database-agent-runs

LibreDB Agent Benchmark

8,199 agent runs · 39 open-weight models served locally, plus one hosted model as a control · 6 task surfaces · 110,711 ledger events · 14,008 refused tool calls

This is the complete measurement record behind the paper What Stops a Small Language Model From Driving a Database Agent. It is not a scored summary: it is every event the server wrote while the runs happened, released so that every number in the paper can be recomputed, and disagreed with, without re-running anything.

The headline finding is that capability is the smallest failure class. Of 2,194 losses by a named model, only 17.0% are runs that never engaged a tool. The other 83% did the work and lost it at the interface.

What was measured

LibreDB Studio ships an agent mode that performs read-only database work through a fixed tool contract, with a server-side verifier that decides whether each run answered its objective. Six surfaces were driven: five in agent mode (investigation, query-optimization, database-assessment, operations, data-analysis) and one in planning mode. Each has its own objective, tool set and verifier.

Every claim a report makes must cite an artifact the run actually produced: the correlation id of a completed read, or the fingerprint of the schema snapshot it captured. Citations are checked against the run's own ledger and refused if they do not resolve. This anti-hallucination rule is also a major source of loss, and refusals.jsonl is where that shows.

Models were served locally through Ollama 0.33.0 on an Apple M5 Max with 64 GB of unified memory, against a fixed synthetic employee/department/salary SQLite schema. One hosted model (gemini-3.5-flash-lite) is included as a control. No real or customer database was connected at any point.

Files

events.jsonl is the evidence; runs.jsonl and the loss taxonomy are an interpretation of it. That separation is deliberate: swap classify() in the exporter and the corpus reclassifies without the data changing.

runs.jsonl fields

Absolute timestamps are not published, since when this machine ran a sweep is not part of the finding, but order preserves sequence, which is.

The protocol, and how to score it

The unit is a cell: one model on one surface, five consecutive runs against a fixed objective. Five consecutive passes locks the cell. A model is complete at 30/30: six cells × five runs. Scores are reported as the pair (27/30), never a percentage, because the denominator is what makes two readings comparable.

python3 score.py runs.jsonl --mode row      # the protocol
python3 score.py runs.jsonl --mode pooled   # the naive reading, for comparison

score.py has three modes and the gap between them is the most important thing in this benchmark:

• —`pooled`: any five-in-a-row anywhere in a cell's history. The obvious implementation.
• —`session`: the five passes must fall inside one contiguous sitting of that cell. Fixes pooling within a cell.
• —`row` (default, and the protocol): all six cells must lock inside one sitting of the model. Fixes pooling between cells, which is where the error actually lives.

Measured on this corpus, pooled and session both report llama3.1:8b and cogito:8b at 6/6 and 30/30. row reports 23/30 for both. Re-reading those two models by hand, six surfaces and one sitting, returned 24/30 and 21/30. The looser readings overstate by seven points, and they do it while every underlying cell genuinely is 5/5. Six cells locked on six different days is six results, not one row, and a row is what a deployment gets.

If you publish a number from this dataset, publish the row number.

Leaderboard (--mode row, models with ≥30 runs)

* marks a locked cell. inv/que/dat/ope/dat/pla are investigation, query-optimization, database-assessment, operations, data-analysis, planning.

Full output for all 31 qualifying models: run score.py, or see leaderboard-row.txt.

This leaderboard is a property of the corpus, not of the product. LibreDB Studio ships 35 models verified at 30/30 under their own measured per-model settings; most of those verification sittings predate the ledger fields this dataset needs, so they do not appear here at 6/6. Read this table as "what these 8,199 runs support", which is the only thing a released dataset can honestly claim.

The failure taxonomy

Each loss is classified by what the ledger shows the run did (classify() in hf-export.py). The table below is every model-attributed loss, all modes, which is the paper's secondary figure. The paper's headline table is agent mode only (2,100 losses: transport 36.2%, clock 25.8%, verification 20.7%, capability 17.3%), because planning mode invokes no tool and so cannot produce a transport loss. anc/verify.py checks both.

Pass rate by surface (named-model runs):

The ordering tracks how many schema-constrained artifacts a surface requires, not how hard its question is. The two hardest both demand a second structured deliverable beyond the report.

Refusals concentrate hard: compose_report accounts for 9,707 of 14,008, 69.3%, of which 6,243 are INVALID_TOOL_INPUT and 3,464 UNVERIFIABLE_EVIDENCE. The one tool that turns completed work into a delivered answer is where seven refusals in ten happen.

Known limitations: read these before using the numbers

1. 1.Unbounded context is a memory confound, and this corpus contains it. With no OLLAMA_CONTEXT_LENGTH set, Ollama admits each model at its full advertised window. mistral-nemo:12b, 7.1 GB on disk, was resident at 51 GB with a 262,144-token context on a 64 GB machine; free memory fell to 6%. Runs taken in that state are not measurements: one granite4.2:3b planning cell read 0/5 with all five runs hitting the 90-second turn limit having invoked no tool, and read 5/5 with the context capped at 32,768. All figures here were taken before the cap, so treat them as a lower bound with unknown per-model bias: models with larger advertised contexts were penalised more. Anyone benchmarking local models should check ollama ps and its CONTEXT column; note that Ollama.app reclaims port 11434 and launchctl setenv does not reach it, so a capped ollama serve can silently fail to bind while the uncapped server answers.
2. 2.Unequal sampling. Runs per model range from 20 to 629 and were allocated by operational interest, not by design. This is not a randomised comparison. The taxonomy shares over 2,194 losses are the more robust figure; per-model rates have very unequal confidence.
3. 3.3,248 runs carry no model. They predate the driver-resolved ledger event. They are in events.jsonl and excluded from every per-model figure. Whole-corpus and attributed-only taxonomy shares agree within 2.7 points on every class.
4. 4.The `worktree` field is a code version. Runs in different worktrees executed against different builds, including builds before and after five server-side fixes this work produced. Pooling across it silently mixes conditions.
5. 5.One machine, one schema, one agent implementation. The surface ordering and the taxonomy are properties of this tool contract. A contract with fewer structured artifacts per surface should show a smaller transport class, a testable prediction, not a caveat.
6. 6.The 450 ms run deadline was never raised. deepseek-r1:8b's query-optimization cell passes at 443 s and 444 s and fails at the deadline. Raising it would likely close the cell; it is part of the shipped product configuration, and a model that only passes outside it has not been shown to work for a user.
7. 7.Models refused before the first turn are a distinct phenomenon. olmo-3:7b, phi4:14b, command-r:35b and falcon3:10b are refused by the inference server with HTTP 400 … does not support tools; qwen2:7b's capability probe disproves schema-valid tool arguments. Their planning cells still pass 5/5, since planning needs no tools. Scoring these as 0/30 without distinguishing refused at the transport from tried and failed reports two different things in one number.

Privacy

The ledger is server-authored. The export drops any field named like a credential, and the released files were scanned for key patterns, bearer tokens, private-key blocks, filesystem paths and email addresses. The only match is johndoe@email.com, a row in the synthetic sample database the agent read during a run.

Reproducing

hf-export.py, in this repository, builds all three files from the raw ledgers. The ledger format is one framed JSON object per chunk file: byte 0 is a marker, the JSON begins at byte 1, and the event is under .event.

Citation

@misc{libredb2026dataset,
  title     = {Database agent runs: 8,199 tool-calling runs across 40 language models},
  author    = {Bozoğlan, Cevheri and Gündoğdu, Yusuf and Kaya, Abdullah and Şirin, Koray},
  year      = {2026},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/10485},
  url       = {https://doi.org/10.57967/hf/10485}
}

The paper this dataset accompanies:

@misc{libredb2026paper,
  title         = {What Stops a Small Language Model From Driving a Database Agent},
  author        = {Bozoğlan, Cevheri and Gündoğdu, Yusuf and Kaya, Abdullah and Şirin, Koray},
  year          = {2026},
  eprint        = {2609.21341},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2609.21341},
  note          = {Preprint. Every figure regenerates from this dataset with \texttt{verify.py}}
}