Datasetpublic

samuki-hf/ocaml-reference-solutions

OCaml reference solutions for LiveCodeBench OCaml reference solutions for the multilingual-LiveCodeBench OCaml problem set, together with the list of problems the harness cannot score. 436 problems in the universe 415 are scoreable, and each has a verified reference 21 are excluded as unscoreable (see the errata config) Configs references is one row per problem. errata lists the problems the harness cannot score, with the evidence for each. from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/samuki-hf/ocaml-reference-solutions.

sourceHugging Faceupdated 7d agoView on Hugging Face

0likes53downloads

Dataset Card

OCaml reference solutions for LiveCodeBench

OCaml reference solutions for the multilingual-LiveCodeBench OCaml problem set, together with the list of problems the harness cannot score.

—436 problems in the universe
—415 are scoreable, and each has a verified reference
—21 are excluded as unscoreable (see the errata config)

Configs

references is one row per problem. errata lists the problems the harness cannot score, with the evidence for each.

python

from datasets import load_dataset
refs = load_dataset("samuki-hf/ocaml-reference-solutions", "references", split="train")
errata = load_dataset("samuki-hf/ocaml-reference-solutions", "errata", split="train")

What `verification` means

Every solution was compiled with ocamlopt and run against the real hidden tests.

value	meaning	count
`exact-match`	passes every official test byte for byte	410
`float-tolerance`	correct; differs only in float formatting	5
`validated-not-exact`	proven correct; the problem accepts several answers	12
`unverified`	from LiveCodeBench's errata list, not independently checked	8

The float-tolerance rows are correct solutions. Those problems accept an absolute or relative error of at most 1e-6, and no comparator in the LiveCodeBench lineage implements that rule: the original grades per line under Decimal equality, Multi-LCB adds bool aliasing and math.isclose(abs_tol=1e-5, rel_tol=0), and Agnostics compares whole outputs under rstrip().

Why problems are excluded

problem	reason	source	evidence
`abc337_e`	interactive	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc343_a`	multiple-solutions	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc343_e`	multiple-solutions	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc350_c`	erroneous-tests	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc355_e`	interactive	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc362_c`	multiple-solutions	livecodebench	documented in LiveCodeBench's own ERRATA.md
`abc363_f`	multiple-solutions	this-work	palindromic product string; 3000/3000 consistent
`abc366_g`	multiple-solutions	this-work	XOR labelling; 300/300 valid on random graphs
`abc373_g`	multiple-solutions	this-work	non-crossing matching; 300/300 valid matchings
`abc392_f`	erroneous-tests	this-work	2 of 42 hidden tests violate the stated 1 <= Pi <= i (P1 = 3 and P_1 = 5) and their expected outputs contain zeros, though the answer is a permutation of 1..N.
`abc396_e`	multiple-solutions	this-work	ties in the per-bit choice; 400/400 minimum sums
`abc397_d`	multiple-solutions	this-work	x^3-y^3=N can admit several pairs; 0 wrong over N<=200000
`arc181_c`	multiple-solutions	this-work	0/1 grid; 200/200 valid on random P,Q
`arc183_d`	multiple-solutions	this-work	asks for any maximum-score removal order, and the two vertices of an operation may be printed in either order, so even the N=4 test with a unique optimal procedure has four correct printings. Exact match accepts one.
`arc185_c`	multiple-solutions	livecodebench	documented in LiveCodeBench's own ERRATA.md
`arc188_c`	multiple-solutions	this-work	any valid confused set; 250/250 (170 valid, 80 correct -1)
`arc189_a`	erroneous-tests	livecodebench	documented in LiveCodeBench's own ERRATA.md
`arc190_a`	multiple-solutions	this-work	any minimum-cost operation list is accepted; ours is a valid optimum on 43 of 44 hidden tests but byte-identical on only 13. Test 14 is also wrong: it contains the interval [1,10] with N=10, so the minimum cost is 1 not 2.
`arc191_c`	multiple-solutions	this-work	(A,M) with multiplicative order N; 59/59 verified
`arc192_b`	erroneous-tests	this-work	1 of 43 hidden tests is wrong: test 20 (N=10, A=[7,16,3,3,9,8,16,12,8,3]) expects Snuke but the true answer is Fennec. Exhaustive game search agrees with the expected output on all 25 other searchable tests.
`arc195_c`	multiple-solutions	this-work	asks for any valid cyclic placement. Our construction is valid on every case checked and its Yes/No agrees with exhaustive search for R+B <= 5, yet it fails all 5 hidden tests because it prints a different valid placement.

livecodebench entries are documented in LiveCodeBench's own ERRATA.md. The harness does not read that file, so these reach every consumer of the dataset. this-work entries were established here by grading a verified reference against the official tests.

Excluded problems still carry a solution, apart from arc183_d. The verification column records how far each was checked. For the two interactive problems the solution is included for completeness and can only be checked against a live judge.

Feeding stdin

Append a trailing newline to each test input if it lacks one, as the vendored Multi-LCB executor does. Otherwise OCaml Scanf formats ending in a newline raise End_of_file on the last line and a correct program fails every test.

Provenance

provenance	model	solutions
model-sample	qwen3-8b-think	239
agent	sonnet	153
agent	opus	17
model-sample	qwen3-14b-think	17
agent	fable	4
model-sample	qwen3-14b-nothink	3
model-sample	llama31-8b-instruct-nothink	1
model-sample	qwen3-8b-nothink	1