qleap/Knowledge_distilled_dataset_by_NAGISA_V4

NAGISA_V4 teacher shards

Training data distilled from self-play of the generator attic-gensfen reading the NNUE weights NAGISA_V4 (HalfKA-2304), in the shape the trainers read directly. The engine played itself at depth 9 and MultiPV 5, recording the root score and the candidate moves at every ply; manaka-teacher turned that corpus into raw MPK1 streams, and manaka-pack folded identical positions into one row each and wrote these parquet shards.

• —Rows: 1,221,920,925 (records before folding: 1,372,843,225)
• —Shards: 49 (data/teacher-00000.parquet …)
• —Total: 96,940,521,668 B

Each shard holds 25,000,000 rows; only the last one carries the remainder of 21,920,925. zstd compression, 8192-row row groups.

How this was generated

This section is the only source for the settings below. They cannot be recovered from the files: the parquet footer states the conventions the converter applied, not the conditions the games were played under.

The side that played the games

Not all games start from the same position. The deterministic opening left too many duplicate boards before ply 15, so partway through generation the starting position was switched to a book of ply-15 positions. The corpus rows carried the actual opening in a start_sfen column; that column does not survive into these shards, which hold folded positions rather than games.

Generation ran in two passes. The first pass resigned at 3000 cp and carried a score ceiling, and it did not write records whose evaluation exceeded that ceiling. A recovery pass (--recover-from) replayed each game from its first missing record with resignation disabled, recorded through to mate, and rewrote the outcome with the replayed ending. This is why nearly every game ends in mate rather than resignation.

`FV_SCALE` is what makes a centipawn a centipawn. The raw value an NNUE accumulates becomes a score only after division by this number. Read the same weights at the engine default of 16 and every cp here would be 1.75× larger. Before mixing this data with anyone else's, check that this number agrees; teacher data blended across two scales trains on numbers belonging to neither.

What the corpus lost on the way in

manaka-teacher read 1,419,944,833 corpus rows and wrote 1,372,843,225 records, refusing 47,101,608 (3.32%):

`move_not_candidate` is the random moves, and refusing them is the point. A row whose played move sits outside the MultiPV list carries no score for the move that was actually made, so there is nothing to put opposite it in a policy. The generator makes five such moves per game on purpose, to spread the openings; they are diversity in the game record and noise in a policy target.

The 30 malformed_record rows are MultiPV lists that rank the same move more than once — a generator artefact rare enough that a whole corpus holds a handful. They are refused because the fold averages cp over the records that listed a move, and a move listed twice in one record would be counted as two searches when only one happened.

The side that converted them

The conversion follows the conventions in manaka-common::card, which both engine families read through, so teacher data built from one corpus is loss-comparable across them.

The mate threshold is declared by the stream, not assumed by the reader. Mates are stored as ±(32000 − mate distance), and ordinary evaluations are clamped at ±32000, so the magnitude alone does not separate them — the two occupy different bands, and the width of the gap is a property of the generator. The stream header carries that width as mate_slack, and it reaches every shard's footer: 100 here, against V3's 320. A reader takes the threshold as mate_scale − mate_slack rather than hard-coding either number.

The 1512.173 in `value_q` is dlshogi's 756.0865 doubled. Theirs is calibrated for a [0, 1] sigmoid; putting the same win-rate curve through a tanh into [-1, 1] doubles the coefficient.

Mate positions skip the softmax. Where a mate has been resolved the best move sits tens of thousands of centipawns above the rest, and a softmax over that is not a distribution in any useful sense.

Schema

There is no legal-move list and no legality mask. A reader that needs either regenerates it from packed.

There is no outcome column

Earlier revisions of this dataset carried a `value_z` holding the result of the game each position came from. It has been removed, and the column is absent rather than emptied.

This is teacher data: positions with an evaluation attached. Nothing in that definition says a game was played through them. The fold below is the plain case — the key is the board alone, so one row here is every game that ever reached that board, and asking which of them it "came from" has no answer. The initial position alone folds 226,572 records; a mean over their results is a number about no game in particular.

Absence is the only spelling a reader cannot mistake for a measurement. 0 reads as a draw. NaN survives every arithmetic it touches, including the (1 − ratio)·z + ratio·q blend that is meant to switch the outcome off at ratio = 1. A column that is not there is read back as "not stated" — the reference trainers give None and Option::None — and the value target is value_q whole.

Nothing else changed. The rows, their order, their row groups and every other column are what the previous revision held; the earlier revision remains reachable by its commit if you want the column back. Self-play corpora, whose positions were played through, do carry an outcome.

What packed holds

96 bytes, manaka_core::pack. The position itself — neither engine's input layer: Manaka's accumulator and DLManaka's input planes are both built from it at load time.

byte
 0..81   the board, one byte per square, in square-index order
81..88   Black's hand, one count per piece kind
88..95   White's hand, same order
    95   side to move, 0 Black / 1 White

Square 0 is 9a and square 80 is 1i: square = (9 − file) × 9 + (rank − 1). A board byte carries the colour in bit 4 (0 Black / 1 White) and the piece kind in the low nibble; an empty square is 0xFF. Hands count pawn, lance, knight, silver, gold, bishop, rook, one byte each.

A candidate is a move, not a label

mv is a move16 — chisaki's move representation, verbatim.

 bit  15 │ 14 │ 13 ──────── 7 │ 6 ──────── 0
      ✗  │ pr │ from or 81+kind │     to

to in bits 0–6, the origin in bits 7–13, promotion in bit 14. A drop puts 81 + hand-index in the origin field, so the dropped kind is readable from the move itself.

Both the policy label and the moving piece derive from the bits alone, so no shogi library is needed to read the shards.

The candidate count is the MultiPV width, and it is countable. A row lists the moves the search reported, so a reader that wants the width reads the list rather than trusting a number declared elsewhere. Measured on shard 0, the width runs 1 to 48 with a mean of 4.798:

Below 5 is a position with fewer legal moves than the MultiPV width. Above 5 is the fold, where two searches of the same board listed different moves and the union kept both.

Folding

An identical position is one row. The fold key is the 96 packed bytes alone — not (packed, ply): whether the game reached the board in 40 moves or 48, a search is handed a board, not a route. Records that met the same board through transpositions or across games merge as:

• —value_q — the mean over every folded record
• —candidates — the union, matched by mv; cp is the mean over the records that listed the move
• —ply — the minimum
• —weight — the record count; 1 means the position occurred once

prob is computed once, after the fold: a softmax over the mean cp at a temperature of 100 centipawns. Rows containing a mate score (|cp| ≥ 31,900) are one-hot on the leading candidate instead.

The fold removed 150,922,300 rows, 10.99% of the records. Measured on shard 0, 98.66% of rows have weight 1 and the mean is 1.1146 — but the maximum is 226,572, which is the initial position: every game that started from startpos rather than from the ply-15 book passed through it.

A move missing from a teacher record is a cut, not a zero. MultiPV reports the top n; a move below the width was ranked worse than the worst listed move, which is an upper bound and not a measurement. Folding A, B, C with A, B, D therefore puts C and D into the same softmax on the strength of one search each, while the other search's opinion of them is discarded. Moves seen once come out slightly over-weighted. What bounds this is how often records meet at all — measured on shard 0, 1.34% of rows carry more than one search, so the bias rides on one row in seventy-five and on none of the rest.

Key-value metadata

Every shard's parquet footer states the conventions the converter applied. Check them before reading.

The mate threshold is ±(mate_scale − mate_slack); value_q is the score through tanh at eval_coef.

`policy_source` is also what says there is no outcome column. softmax_cp means the targets came from a search run on a board; visits means a game was played and its result is the one thing it earned. The writer reads no separate flag for the column, so a corpus cannot be published claiming an outcome it does not have.

Every pair here is copied from the stream header or from the converter's own constants — none of it can be typed by hand. There is no flag for adding metadata, deliberately: a key an operator supplies is a key that can disagree with the rows underneath it, and only that operator's memory would ever catch it. Provenance the stream header has no field for — which generator and evaluation produced the scores — lives in "How this was generated" above, in one place, in prose that can be read against the rest of the card.

`mate_scale` is 32000 here and 30000 in the V3 shards. The two are not interchangeable and the fold refuses to mix them: a mate stored on one scale is an ordinary large evaluation on the other. Check this key before folding these shards together with anyone else's.

Row order

One uniform shuffle over the whole corpus. The fold itself has to be partitioned — it runs one of 128 hash buckets at a time, so that identical boards always meet — but a bucket is a property of the position bytes, and writing the buckets one after another would leave that property standing in the row order. So the folded rows are scattered back across 128 spools chosen uniformly at random, and each spool is loaded and permuted on its own before it is written. Where a row lands depends on nothing about the row.

The permutation is a Fisher–Yates with an unbiased draw, seeded from a constant, so the same streams reproduce the same shards. Deterministic, and still not an order.

No game order, phase or chronology survives. Measured on shard 0: the lag-1 autocorrelation of ply is −0.000140 and its correlation with the row index is −0.000206. Neighbouring rows are not neighbouring plies of a game.

Nor does any block structure. Cut shard 0 into blocks and compare the spread of their mean ply against what independent sampling of that block size would give; a ratio of 1 means nothing survived.

The ratio is itself noisy — its standard deviation is about 1/√(2(blocks−1)), so ±0.15 at 24 blocks and ±0.014 at 2,500 — and every row here sits inside that.

Across the 49 shards the mean ply runs 69.648 to 69.677 against an overall 69.6633, on a standard deviation of 36.685. A shard mean has a standard error of 0.0073 at 25,000,000 rows, so that whole spread is sampling.

The trainer can read the shards as they come; no shuffle on the reading side is required.

How it compresses

Measured over shard 0, per row:

Measured before value_z was dropped; the totals are the sum of the columns still listed. The column it lost cost 0.59 B/row raw and 0.23 compressed, so removing it is a rounding error against the corpus — that a ±1 field packs almost to nothing is a reason it was cheap to keep, not a reason it belonged.

`packed` costs 31.15 bytes a row here against 5.63 in the V3 shards. The bytes are the same 96-byte mailbox in both; what differs is how much the corpus repeats itself. V3's games all began from a 56,326-position book, and zstd found the shared openings. These games begin from startpos or from one of about 2.5 million ply-15 positions, so far fewer boards resemble their neighbours and there is correspondingly less for the compressor to fold away.

prob is the largest column, and it is a float that carries a softmax output — high entropy by construction, and it does not compress.

No quiescence resolution

The positions are the games' positions, as reached — no quiescence search was applied to replace a position with a quiet one before recording. A row can sit in the middle of a capture sequence, and its cp values are the generator's MultiPV search scores of that position itself. A consumer that wants quiet-only positions has to resolve or filter them on its own side.

Train/val

There is no split column, and none is needed. Every row is a unique position — the fold key is the 96 packed bytes, so no position can appear on both sides of any cut — and the rows are uniformly shuffled, so slicing off any fraction is a valid random split at position granularity, the usual practice for NNUE training data. Take it from the tail, the head or the middle.

The raw streams behind this

The generator's unprocessed output — MPK1 records as attic-gensfen appended them, before manaka-teacher refused anything and before any position was folded — is published separately:

ds = load_dataset("qleap/Raw_dataset_by_NAGISA_V4", split="train")

These shards cannot be rebuilt from an already-folded source: the fold has to see every record to count weight and to average cp.