Honkware/catbench-results

CatBench results

Model outputs for CatBench, a small benchmark that asks a model to draw a cute kitten two ways and looks at what comes back. Produced by the /catbench command in blockquant.

Upstream publishes its own results at Katehuuh.github.io/demos/CatBench/assets. This dataset holds runs for models that are not in that set. /catbench checks both and only rents a pod when neither has the model, so the two do not duplicate each other.

The prompts

Verbatim, unchanged between runs:

1. 1.Create a detailed SVG image of a cute kitten.
2. 2.Write a Python script that draws a cute kitten using matplotlib.

Layout

One entry per benched model, keyed by the repo stem lowercased with _ and whitespace folded to -, matching upstream's key format.

manifest.json                 index of every entry
assets/<key>-svg.jpg          the SVG, rasterized
assets/<key>-python.jpg       what the matplotlib script drew
assets/<key>.svg              raw SVG the model emitted
assets/<key>.py               raw script the model emitted

Each manifest entry records the model id, both image paths, both source paths, which loader ran (exl3 or transformers), the engine version, and the run date.

What is here and what is not

Only runs where both halves produced something. A run whose script crashed, whose SVG came back empty, or where the model wrote prose instead of code is not stored, so those can be retried after a fix rather than being cached as a result.

Raw sources are kept alongside the images because the rendering is lossy: a script that draws a kitten badly and one that draws nothing look more alike as a jpg than they do as code.

Caveats

Greedy decoding, one sample per prompt, no retries. This measures what a model does on one attempt at two prompts. It is a curiosity benchmark, not a capability evaluation, and it should not be read as one.