jeorgexyz/lua-agent-evals

Lua Agent Evals

The evidence behind lua-agent-lab, from lua-agent. The experiment separates structural tool-call validity from useful tool selection and complete task success.

Contents

The test split is an inspectable demonstration suite, not a training set or a representative benchmark. Do not treat 30 trials as 30 independent problems: ten prompts are repeated with 0, 1 and 2 trailing spaces, matching the original ablation. No confidence interval or population-level claim is implied.

Collection and provenance

Generated with python -m space.export_dataset --tiny --qwen from the source repository. Each measured row includes its UTC collection time, a SHA-256 digest of the relevant source, exact settings and pinned model revisions. manifest.json records the checksum and row count of each data file. The baseline source commit was 1738e7dd63bf0d65a055fd11a497c30b3f47eff4; the adapter and metadata additions are represented by the per-row source digest. provenance/source/ contains the exact normalized source and fixtures covered by that digest, so the exported runs remain auditable without a moving branch.

TinyStories runs backend/local.lua and grammar.lua inside Lua 5.4 with PyTorch float32 numerical kernels from tools/llama2c.py. The tokenizer is the original Lua implementation. This differs numerically from the original pure-Lua inference, so new rows are not attributed to the historical result. Both arms scaffold THOUGHT: and CALL:. Constraints can append a final grammar completion at the token limit; forced_finishes counts these cases, and each affected step has forced_finish: true.

Qwen uses the official pinned GGUF repository and llama-cpp-python 0.3.16 on CPU. Constrained decoding uses a schema describing the available tools; free decoding omits that schema. Prompts and greedy sampling are otherwise the same. The runtime scaffolds CALL: in both arms and does not repair free output. These are distinct from the historical Ollama Qwen run.

Scripted rows use the original fixed turn lists and actual Lua loop, with fresh virtual fixtures per run. They demonstrate control flow, not model capability. The large_context_budget variant raises the budget to 100000 estimated tokens so eviction does not occur; it does not simulate overflow. The approval-gate ablation only permits virtual writes. drop_oldest changes the eviction policy; no_error_recovery removes both parse and tool recovery.

The historical 30/30 constrained versus 0/30 free result, with 0/30 usable tool selections in both arms, is preserved verbatim in provenance/constrained-vs-free.txt. Complete historical trial traces were not saved; none have been invented. provenance/eval-qwen1.5b.txt likewise preserves the historical 6/8 Ollama report without implying complete raw traces exist.

Schema and metrics

The three run configurations use the same flat schema:

• —prompt, task_id, expected_tools, expected_outcome: the task and labels.
• —selected_tools, answer, reason, passed, judge_note: observed outcome.
• —generated_turns, valid_calls, tool_call_validity: parsed tool calls / generated turns, in percent. answer is a tool; argument correctness and whether a tool is registered are not checked by this syntax metric.
• —first_tool, correct_first_tool: first generated turn's choice against accepted tools. A malformed first turn is incorrect. Unlabeled or unavailable choices are null; the directory-listing decoder prompt has no available suitable tool in the two-tool catalogue.
• —steps, malformed_calls, loop_detected, context_evicted, forced_finishes: measurements from the step records.
• —settings_json, trace_json, approvals_json: JSON-encoded structured records. Strings preserve arbitrary tool argument shapes without a lossy Arrow union. Decode with json.loads(row['trace_json']).
• —model, inference_kind, decoder, token_accounting, elapsed_seconds, run_at, source_sha256, model_revisions_json: provenance and conditions.

Task success uses the repository's solved functions for real models and check functions for scripts. Scripted checks require the intended mechanism to fire. Some judges are substring based and can accept spurious matches; read traces rather than treating the score as a strong capability estimate. First-turn decoder trials have passed: null because they do not run a full task.

Use

import json
from datasets import load_dataset

rows = load_dataset('jeorgexyz/lua-agent-evals', 'decoder_trials', split='test')
trace = json.loads(rows[0]['trace_json'])

This dataset contains only developer-run experiments on synthetic tasks and repository fixtures. Public Space visitors' prompts and traces are not collected or uploaded. It is intended for reproducibility, teaching and debugging, not training on private agent conversations. All source and authored fixtures are MIT licensed; the original model repositories document their checkpoint licenses.