qleap/Training_dataset_by_NAGISA_V4

NAGISA_V4 ply-37 teacher shards, games played to the end

Self-play of attic-gensfen reading the NNUE weights NAGISA_V4 (HalfKA-2304), in the shape the trainers read directly. Every game starts from a balanced ply-37 position, makes no random moves, and runs until it actually ends. Identical positions are folded into one row each.

• —67,108,864 rows — exactly 2^26
• —16 shards of 4,194,304 rows, 512 row groups each, zstd, 5,208,620,326 B total

Against `Opening_dataset_by_NAGISA_V4`, which cut its games off at 3000 centipawns and made five random moves each: this one has a `value_z` outcome column, and mates are in the data (5,985,285 rows at |value_q| = 1, 8.92%).

How it was generated

Every node ran this; only --out and --seed differed.

attic-gensfen \
  --eval-dir <NAGISA_V4 weights> \
  --start-sfens <balanced ply-37 openings> \
  --depth 9 --multipv 5 --fv-scale 28 --format pack \
  --max-ply 512 --entering-king CSARule27 \
  --random-move-count 0 --eval-limit 0 --resign-cp 0

Version 9.70.0 (1fc9e081-dirty), eight nodes, 256 threads and 16 MB of hash per node. Four of the flags carry the character of the corpus:

• —`--fv-scale 28` is NAGISA_V4's calibration. Read the same weights at the engine default of 16 and every cp here would be 1.75 times larger.
• —`--random-move-count 0` — always the search's own move. Measured over 10,882,623 records, the played move is the first candidate in all but two, and is never some other candidate.
• —`--eval-limit 0 --resign-cp 0` — nothing stops a game on its score.
• —`--max-ply 512` counts plies played, so games opening at move 37 reach move 548.

The openings are 6,045,847 balanced ply-37 positions, drawn with replacement: 669,540 games drew 633,795 distinct ones and produced 68,155,885 records. Every game opens at move 37, so every opening position is Black to move.

How the games ended

From Black's point of view: 351,641 Black wins, 316,854 White wins, 1,045 draws.

What was lost on the way in: 22 records of 68,155,885 were refused as malformed MultiPV lists (the same move ranked twice), and the converter's own cap at move 512 dropped 2,262 more and relabelled 71 games as draws.

Schema

ply is a real move number taken from the sfen, not an offset: it runs 37 to 512, mean 92.2203, standard deviation 38.2549.

value_q is tanh(cp / 1512.173), and ±1 for a mate score. The positions are as the games reached them — no quiescence search was applied, so a row can sit in the middle of a capture sequence.

value_z, and what folding does to it

+1 won, −1 lost, 0 drawn, from the side to move's point of view. After the fold it is a mean, because a position reached by several games becomes one row: two wins and a loss give +0.333.

A zero is either a drawn game or outcomes that cancelled, and nothing in the row separates them. `weight = 1` selects the rows that state one game's result whole, which is 99.50% of them.

The reference trainers blend the two value targets as (1 − q_ratio)·value_z + q_ratio·value_q, q_ratio defaulting to 0.5. On a shard with no value_z the trainer takes value_q as the whole target instead, so the same config fits a different thing here than it does on the opening corpus.

packed

96 bytes, manaka_core::pack — the position itself, neither engine's input layer.

byte
 0..81   the board, one byte per square, in square-index order
81..88   Black's hand, one count per piece kind
88..95   White's hand, same order
    95   side to move, 0 Black / 1 White

Square 0 is 9a and square 80 is 1i: square = (9 − file) × 9 + (rank − 1). A board byte carries the colour in bit 4 and the piece kind in the low nibble; an empty square is 0xFF. Hands count pawn, lance, knight, silver, gold, bishop, rook.

candidates

mv is a move16, chisaki's move representation, verbatim: to in bits 0–6, the origin in bits 7–13, promotion in bit 14. A drop puts 81 + hand-index in the origin field. Both the policy label and the moving piece derive from the bits alone, so no shogi library is needed to read the shards.

prob is a softmax over cp at a temperature of 100 centipawns, computed after the fold; it is one-hot on the leading candidate when the position is a mate.

The list is the MultiPV width as reported: 89.18% of rows list 5. Fewer means the position had fewer legal moves — common here, because these games run into mating nets. More (0.43% of rows, up to 30) is the fold, where two searches of the same board listed different moves and the union kept both.

A move missing from the list is a cut, not a zero. It was ranked worse than the worst listed move, which is an upper bound and not a measurement.

Folding

The fold key is the 96 packed bytes alone, not (packed, ply): a search is handed a board, not a route. Merged records take the mean of value_q, value_z and each candidate's cp, the union of the candidates, the minimum ply, and the record count as weight.

68,153,601 records folded to 67,530,963 rows, a removal of 0.91%. The ply-37 openings are why so little folds; the largest single source of duplication is the opening pool itself, drawn with replacement. weight reaches 795.

Key-value metadata

Every shard's footer states the conventions the converter applied. It is copied from the stream header and from the converter's constants — there is no flag for writing it by hand.

The mate threshold is ±(mate_scale − mate_slack), so 31,900 here. A shard without `states_outcome` either has no outcome column or has one that states nothing, and mate_scale is 30000 in the V3 shards — a mate on one scale is an ordinary large evaluation on the other, so check both before folding these together with anyone else's.

Row order, and the trim

One uniform shuffle over the whole corpus, not shuffled blocks laid end to end: the fold ran in 128 hash buckets and its rows were scattered back across 128 spools at random, each permuted before it was written. On shard 0 the lag-1 autocorrelation of ply is 0.000313 and its correlation with the row index is −0.000875, so nothing of game order survives. No shuffle on the reading side is required.

The fold produced 67,530,963 rows and the 422,099 past 2^26 were dropped. The shuffle runs before the shards are cut, so this is a uniform random sample, not a slice of anything.

There is no split column. Every row is a unique position and the order is random, so any contiguous fraction is a valid validation split.

The generator's pack streams and the intermediate MPK1 streams are not published; these shards cannot be rebuilt from an already-folded source, because the fold has to see every record to count weight and average cp.