qleap/Training_dataset_qsearched_by_NAGISA_V4

NAGISA_V4 teacher shards, moved to their quiescence leaves

Every record of `Training_dataset_by_NAGISA_V4` walked to the end of its quiescence variation, the deep search's value kept there, and a policy fitted at the leaf itself.

The parent's positions are as its games reached them, with no quiescence search — a row can sit in the middle of an exchange, where the evaluation swings by a piece depending on whose turn it is to recapture. A value fitted on those learns the swing. This corpus is the same searches read at quiet positions instead.

• —51,447,715 rows
• —13 shards of 4,194,304 rows (the last has 1,116,067), zstd, 512 row groups each (the last has 137), 46,244,656,373 B total
• —Both heads. value_q is the parent's number carried across; candidates is a policy measured at the leaf, over every legal move. There is no value_z column, which is explained below.

How it was made

Four steps, all in `rshogi/manaka`.

1. `manaka-unpack` turns the published shards back into the MPK1 record streams the tools read. Two columns do not survive the crossing.

• —`weight` becomes repetition. A row that folded w searches is written w times, so the fold at the far end restores the same weight and the same means. 67,108,864 rows expanded to 67,727,556 records.
• —`value_z` is dropped. A stream's value_z is one game's outcome, so it is +1, 0 or −1 and nothing else; a shard's is the mean over the searches that folded into the row. The mean of two games is not a game, and 0.0 already means a draw, so there is no value to write that is not a claim about a game nobody played.

2. `attic-qsearch` runs the quiescence search of `attic-shogi` 1fc9e08 on every record, reading the same NAGISAV4 weights (HalfKA-2304) at the same `fvscale 28, and plays the returned variation out. The leaf is stored. The value is the one the record arrived with, sign-flipped when the walk took an odd number of moves, because these are stored from the moving side's point of view and an odd walk hands the move to the opponent. This is upstream's qsearch_psv` on this repository's stream format.

attic-qsearch <unpacked streams> \
  --out <leaf streams> \
  --eval-dir <NAGISA_V4 weights> \
  --value record --jobs 20 --hash-mb 64

Mean walk 1.67 moves over the records that moved; the longest was 25.

3. `manaka-pack` folds and shards, the same way the parent was built: identical positions collapse to one row, and the whole corpus is permuted before it is cut. 67,727,556 records folded to 51,447,715 rows, a removal of 24.04% — against 0.91% in the parent, because many positions share a leaf. That is the whole point of the exercise, and it is also why this corpus is smaller than the one it came from.

4. `attic-qsearch --mode policy` reads the folded shards back and writes the candidates column, leaving packed, ply, value_q and weight byte for byte as step 3 left them. It is described in the next section.

The policy

The parent's candidates could not be carried across: the move a record stored was the best move at the position it stored, and that position is gone. So the policy here is not the parent's, moved — it is measured at the leaf, and it is the one column of this corpus that is genuinely new.

For each row:

1. 1.Every legal move is enumerated at the leaf. No candidate list, no MultiPV cut, no move ordering heuristic decides what gets in.
2. 2.Each move is played and the same NAGISAV4 quiescence search runs on the child, at the same `fvscale 28 and the same layer-stack table (progress8ek`) as steps 2 and 3.
3. 3.The child's score is negated back to the row's side to move and stored as that move's cp.
4. 4.prob is softmax(cp / 100) over the move list, which is the softmax_cp contract the metadata already declared. When the best cp reaches the mate band (±31,900), the distribution collapses to one-hot on the first maximum instead, so a forced win is not smeared over the moves that also happen to be winning.

attic-qsearch --mode policy <folded shards> \
  --out <policy shards> \
  --eval-dir <NAGISA_V4 weights> \
  --jobs 192 --hash-mb 16

6,245,149,136 moves carry a score, 121.40 per non-terminal row. Every non-terminal row's probabilities sum to 1 within float32 rounding: across the corpus the sums run 0.999997 to 1.000007.

The result is a policy distilled from qsearch evaluations at quiet positions, not a copy of the parent's move choices. It agrees with value_q without being derived from it: tanh(best cp / 1512.173) and value_q correlate at 0.8664 and agree in sign on 81.02% of the decided rows.

Terminal rows have no legal move. They stay in the corpus with an empty candidates list and contribute value loss only. A trainer must exclude them from the policy loss and from policy metrics rather than let an all-zero target dilute the batch mean; ours does this by masking on target.sum(-1) > 0.

Schema

packed and mv are laid out exactly as in the parent — read that card for the byte order and the move16 bits. One cosmetic difference: the struct inside candidates is named element here and item in the parent, which is the Arrow list convention the writer used. The values and their order are unaffected, and readers that address the column rather than its inner field name see no change.

ply follows the position: a walk of n moves adds n, so it runs 37 to 515 where the parent stopped at 512. Mean 91.1049, standard deviation 38.7440.

value_q is untouched apart from the sign: still tanh(cp / 1512.173), still ±1 for a mate score. Mean 0.0391, standard deviation 0.4869. 3,588,473 rows sit at |value_q| = 1, 6.97%.

weight sums to 67,727,556 across the corpus, which is exactly the record count — the round trip through MPK1 conserves the fold. It reaches 795, and 76.71% of rows have weight = 1.

Why there is no value_z

See step 1. Beyond the fold, the leaf is not the position the game passed through, so even an honest outcome would be attached to a board that game never reached. Train the value head on `value_q` alone here. The reference trainers blend the two as (1 − q_ratio)·value_z + q_ratio·value_q; with no value_z column they take value_q as the whole target, which is the right thing and happens without a config change.

Key-value metadata

states_outcome is false here and true in the parent, which is the one field that tells the two apart without reading a row. The mate threshold is ±(mate_scale − mate_slack), so 31,900; it is 30000 in the V3 shards, so a mate on one scale is an ordinary large evaluation on the other.

Row order

One uniform shuffle over the whole corpus, not shuffled blocks laid end to end: the fold ran in 32 hash buckets and its rows were scattered back across 32 spools at random, each permuted before it was written. On shard 0 the lag-1 autocorrelation of ply is 0.000167 and its correlation with the row index is −0.000421. Step 4 preserved that order row for row. No shuffle on the reading side is required.

There is no split column. Every row is a unique position and the order is random, so any contiguous fraction is a valid validation split.