DiogenesChen122/Dr.Sparse-Ornith15-9B-eval-b200-otf-spgemm-partial
Dr.Sparse — Ornith-1.5-9B SpGEMM baseline (partial, 12/81 matrices) Partial baseline of ornith-ai/Ornith-1.5-9B on the Dr.Sparse OTF held-out test set, SpGEMM only, levels 1-3 (level4 excluded). B200, single trajectory (no tree search). Why this run is partial The run was stopped after 12 of 81 matrices. HiPerGator terminates jobs that hold a GPU without using it, and this eval layout gives each matrix its own GPU while the agent spends most of each iteration… See the full description on the dataset page: https://huggingface.co/datasets/DiogenesChen122/Dr.Sparse-Ornith15-9B-eval-b200-otf-spgemm-partial.
Dr.Sparse — Ornith-1.5-9B SpGEMM baseline (partial, 12/81 matrices)
Partial baseline of ornith-ai/Ornith-1.5-9B on the Dr.Sparse OTF held-out test set, SpGEMM only, levels 1-3 (level4 excluded). B200, single trajectory (no tree search).
Why this run is partial
The run was stopped after 12 of 81 matrices. HiPerGator terminates jobs that hold a GPU without using it, and this eval layout gives each matrix its own GPU while the agent spends most of each iteration waiting on a remote vLLM server. Cluster staff cancelled 11 of the 23 finished elements before the run was halted. The 12 matrices here all completed their full 15 iterations and are unaffected.
Configuration
Result
None of the 12 matrices produced a correct kernel that beat cuSPARSE. Across 15 iterations each, no iteration yielded a passing kernel.
The dominant failure mode is that this model writes its chain-of-thought into the response body rather than a separate reasoning channel. Single responses reached 190-225 KB of first-person deliberation containing no function definitions, truncated mid-sentence at the output limit. With a 65536-token window and prompts already consuming roughly 57000 of it, only 8000-12000 tokens remained for code, which the deliberation exhausted. A follow-up run with a 262144-token window is the direct test of that explanation.
Layout
spgemm/<level>/SpGEMM_<matrix>-<timestamp>/
final_state.json full agent state, per-iteration history
kernel_iter_<i>.cu kernel source per iteration
chat_iter_<i>.json full prompt/response transcript
compile_error_<i>.txt nvcc output when compilation failed
verification_feedback_<i>.json
benchmark_spgemm.cu, cusparse_spgemm.cu, Makefile, data_loader.hMatrices: AG-Moniense, Boeingmsc01440, Brunetierethermal, DIMACS10vspmsc10848300sep100in1Kout, DRIVCAVcavity16, DRIVCAVcavity19, FIDAPex3, GoodwinGoodwin023, GsetG4, GsetG55, PajekSciMet, Sandiaadderdcop_38.