aaylmao/memory-machines-inkling-run
Memory Machines model evaluation logbook
Static publication for a protocol-faithful evaluation of OpenRouter conditions on the released Memory Machines flashcard dataset.
Completed conditions:
- Two independent, cache-disabled Auto Router runs
stealth/ox-alphathinkingmachines/inkling-small
Each row uses one direct zero-shot JSON judge call with the released instruction and formatter. There are no tools, retrieval steps, or agent loops. The released boolean label (T3 positive) and the report's conceptual label (T2 + T3 positive) are both disclosed.
Why the queue runner was made
The original JSONL runner was vulnerable to overlapping detached processes. Concurrent writes could duplicate rows or interleave records. The study repository therefore uses SQLite transactions, a single-run lease, idempotent row keys, interrupted-row recovery, retries, and retained dead-letter state.
The runner also fixes a startup cleanup bug: if dataset loading failed before SQLite initialization, cleanup previously emitted a misleading no such table: runner_lock error. Cleanup now verifies the lock table before releasing it.
Auto Router results are a routed mixture, not a named-model benchmark. Named-model extensions are reported separately and are never pooled with the Auto Router repeatability aggregate.