qleap/MPK1_dataset_by_NAGISA_V4

MPK1 dataset by NAGISA V4

Shogi teacher positions generated by attic-gensfen with the NAGISA V4 HalfKA-2304 evaluation function, carried in MPK1 — one record per searched position, before duplicate boards are folded into one. This is the input behind qleap/Knowledge_distilled_dataset_by_NAGISA_V4: no folding, no deduplication, no policy normalisation, no shuffling. 14,101,066 games, 1,372,612,150 positions.

This is not the whole corpus its generator wrote. qleap/Raw_dataset_by_NAGISA_V4 holds that — 1,419,944,833 records in YaneuraOu's 60-byte PackedSfenValue format. Three removals stand between it and this repository: the conversion dropped 47,101,608 records, 3.32%, for a reason given below; 106,568 more were cut off the long tail of the games, 0.008% of what was left; and 124,507 belonging to 329 games the generator never finished were taken out whole, both for reasons given below. If you want every position the generator produced, take the raw repository.

1,024 games do not carry the result they were played to. They are the games the cut truncated, and they are stored as draws; the section on the cut says why and how to find them.

qleap/MPK1_dataset_by_NAGISA_V3 is this same format one generation back. Do not pool the two without reading both headers. They declare different conventions — mate_scale 32000 here against 30000 there, and a mate_slack of 100 against an effective 320. The same cp is an ordinary evaluation in one and a mate in the other, and nothing downstream checks.

Files

`load_dataset` does not read this. MPK1 is a binary record stream, not a format datasets knows; the repository is an archive, and the folded parquet at qleap/Knowledge_distilled_dataset_by_NAGISA_V4 is what a trainer loads. Fetch the files and walk them:

from huggingface_hub import snapshot_download

path = snapshot_download("qleap/MPK1_dataset_by_NAGISA_V4", repo_type="dataset")

# one part, decompressed
unzstd -c part-00000.mpk.zst > part-00000.mpk

The parts do not concatenate. Each one is a complete MPK1 stream and opens with its own 24-byte header, so cat part-*.mpk.zst | unzstd puts a header in the middle of the record run and a reader walks off the framing at that byte. Decompress and read them one at a time; the headers are byte-identical, so nothing is lost by dropping all but the first.

A game can straddle a part boundary. The cut is at a record count, not at a game boundary, and 136 of the 136 joins land inside a game. A reader that needs whole games either reads the parts in name order as one logical stream, or discards the partial game at each end.

Format

Defined in manaka_core::stream. Little-endian throughout.

header, 24 bytes
  magic u32 "MPK1" | version u16 | packed_len u16
  pov u8 | policy_source u8 | mate_scale u16
  fv_scale u16 | mate_slack u16 | eval_coef f32 | flags u32

record, variable length — 12 + packed_len + 8·n_cand bytes
  ply u16 | value_z f32 | value_q f32 | n_cand u16
  packed [u8; packed_len]
  candidates × n_cand : mv u16 | visits u32 | cp i16

There is no record count in the header and no index: a part is read by walking it. There is no legal-move list and no legality mask either — a reader that needs one regenerates it from packed.

What this stream declares

Every part opens with the same 24 bytes:

4d 50 4b 31 02 00 60 00 00 01 00 7d 1c 00 64 00 89 05 bd 44 00 00 00 00

The mate threshold is `mate_scale − mate_slack` = 31900, and the slack is narrow on purpose. This generator's ordinary evaluations run right up against the scale — the largest non-mate score observed is 31753 — so the 320 the V3 stream falls back on would reclassify real evaluations as mates. A reader that substitutes some other slack silently changes which rows are one-hot, and nothing complains.

`FV_SCALE` is what makes a centipawn a centipawn. The raw value an NNUE accumulates becomes a score only after division by this number. Before mixing this data with anyone else's, check that this number agrees; teacher data blended across two scales trains on numbers belonging to neither.

What packed holds

96 bytes, manaka_core::pack. The position itself — not an engine's input layer; a feature space can change without the corpus being rebaked. Note that this is not the 32-byte YaneuraOu packing the raw repository uses.

byte
 0..81   the board, one byte per square, in square-index order
81..88   Black's hand, one count per piece kind
88..95   White's hand, same order
    95   side to move, 0 Black / 1 White

Square 0 is 9a and square 80 is 1i: square = (9 − file) × 9 + (rank − 1), so ranks 1–3 are White's camp and 7–9 are Black's.

A board byte carries the colour in bit 4 (0 Black, 1 White) and the piece kind in the low nibble; an empty square is 0xFF, which cannot collide because no valid byte sets the high bits.

Hands are seven counts each, one byte apiece, and not in the board's kind order — hands run P, L, N, S, G, B, R, the reverse direction. Maxima are 18, 4, 4, 4, 4, 2, 2. This is the one place in the format where the same seven kinds appear under two orderings, so it is worth checking against rather than assuming.

A candidate is a move, not a label

mv is a move16 — chisaki's move representation, verbatim, and the same encoding the raw repository's PackedSfenValue uses.

 bit  15 │ 14 │ 13 ──────── 7 │ 6 ──────── 0
      ✗  │ pr │ from or 81+kind │     to

to in bits 0–6, the origin square in bits 7–13, promotion in bit 14, bit 15 unused. A drop puts 81 + hand index in the origin field instead of a square, so the dropped kind is readable from the move itself.

A reader needs no move generator, and therefore no shogi library. A policy label is a direction and a destination, so a pawn stepping to 7f and a lance running there are the same number, and which piece was lifted is not recoverable from it. A move16 carries the origin outright, so both the label and the lifted piece fall out of bit arithmetic — seven lines of it in any language.

Candidate lists

MultiPV was 5, and the width of a record's list is its own n_cand, not a number declared beside the file:

6,585,469,371 candidate scores in all, 4.80 to a record. Lists shorter than 5 are not truncation: the generator reports fewer when fewer moves had a settled score. There is no record with an empty list — one would contribute zero loss and look like learning.

Over all of those scores the extremes are −31,998 and 31,999.

`cp` is a score, not a probability. No softmax has been applied — the temperature and the mate one-hot are applied at the parquet conversion, so this corpus can be re-softmaxed at a different temperature without regenerating it. cand_cp is also not sorted: the order is the MultiPV rank the engine assigned, so a rank inversion is normal.

Unlike the V3 stream, the root score is here too. In the raw corpus the root cp and the leading candidate's score agreed on 100.000% of records, measured over one chunk of 10,000,053, so the root survives in slot 0 rather than only in the saturating value_q. The V3 corpus aligns its candidates at a shallower depth than its root and has no such property.

Reading it

import struct

FIXED, CAND = 12, 8

def records(path):
    with open(path, "rb") as f:
        head = f.read(24)
        assert head[:4] == b"MPK1"
        packed_len = struct.unpack_from("<H", head, 6)[0]
        blob = f.read()
    i = 0
    while i < len(blob):
        ply, value_z, value_q, n = struct.unpack_from("<HffH", blob, i)
        packed = blob[i + FIXED : i + FIXED + packed_len]
        base = i + FIXED + packed_len
        cands = [
            struct.unpack_from("<HIh", blob, base + CAND * k)[0:3:2]  # (mv, cp)
            for k in range(n)
        ]
        yield ply, value_z, value_q, packed, cands
        i = base + CAND * n

(unzstd the part first, or wrap the file in zstandard.ZstdDecompressor.)

Row order

Games are contiguous runs in ascending ply, in the order they came out of the generator — nothing here was shuffled:

A new game starts at every record whose `ply` does not increase over the previous record's. There is no game id and no delimiter.

Games open at two plies and almost nowhere else. Of the 14,101,066 games: 8,181,835 at ply 1 (58.02%), 5,917,944 at ply 15 (41.97%). The remaining 1,287 are scattered over 32 further values, the largest of which holds 1,025. None of them is a part boundary: all 136 joins land inside a game and were stitched before these were counted.

The plies inside a game are not consecutive. This is where this corpus differs from the V3 stream, which has no holes at all: the 47,101,608 dropped records were removed from the middle of the games they belonged to, so a game's plies ascend but skip. 37,219,625 of the 1,372,612,150 records follow a record more than one ply behind them. A reader that treats ply + 1 as "the next record" is wrong here; the ordering is still total, and only the spacing is not.

Measured over all 1,372,612,150 records:

`ply` stops at 512 because it was cut there. The corpus the converter wrote ran to 653, past a generator whose stated maximum game length is 320; everything above 512 was removed from this repository and the games that reached the cap were relabelled. A positional encoding sized to 512 + 1 covers this corpus, but read the cut before bucketing by ply or training a value head on the high plies — the cut is not uniform in what it left behind.

A random row split is fine. The label is a search score that depends only on the position, so nothing has to be held out at a game boundary: slice off whatever fraction you want for validation.

Value targets

The two outcomes are not balanced. The side to move wins on 50.64% of records and loses on 49.32% — 18,216,213 more wins than losses. Some of that gap is the drop, as arithmetic rather than as a theory: the removed records lost 61.21% of the time (measured below), and taking a loss-heavy set out of a corpus leaves the rest tilted toward wins. It does not account for the whole gap, and what else contributes was not established. If a value head or a calibration step assumes a symmetric outcome prior, it does not hold here.

A zero does not mean the players drew. The generator's game loop leaves by exactly three doors — mated and resign, which write ±1, and max_ply, which writes 0. It implements no repetition rule and no declaration win, so a zero is always a game that ran out of plies, never a game that ended level.

Every zero here is a game [the cut](#the-512-ply-cut) truncated — all 516,611 of them, across 1,024 games, and the drawn column holds nothing else. The generator abandoned 329 further games, and those were removed rather than published, so that a zero carries one meaning instead of two.

Those records are not a tail: they run the whole length of the games they belong to, from ply 1 to the cap. What is confined to the tail is the games — every game still alive at ply 512 is one of them — so the drawn share of records climbs with ply until, at the cap itself, it is everything. That climb belongs to this repository, not to shogi.

Draws are stored as a float zero whose sign bit is sometimes set — -0.0 arrives when a drawn game's record was rotated from White's side. value_z == 0.0 is true for both; a bitwise comparison against 0x00000000 is not. 0 of the drawn records carry the negative zero — fewer than the uncut stream held, because the redraw wrote +0.0 over every record of the games it touched, sign bit included.

The root `cp` is not stored directly, only `value_q`. It can be read back as 1512.173 · atanh(value_q) where the position is not a mate, but tanh saturates in f32 at |cp| ≥ 13627 and a mate's value_q is only the signum. The candidate list is the better source: slot 0 carries the root score unsquashed, as noted above.

What was dropped

manaka-teacher read all 1,419,944,833 raw records and wrote 1,372,843,225 — the count before the cut:

1419944833 rows read, 1372843225 written, 47101608 rejected
  move_not_candidate=47101578  malformed_record=30
  bad_sfen=0  side_mismatch=0  illegal_candidate=0
  unlabelled_move=0  no_legal_moves=0

Essentially all of them are one case: the move actually played was not among the five the search reported. V4's generator plays moves from outside the search by design, and those records are the result.

Such a record is not malformed, and most of it is perfectly good — the position, the MultiPV scores and value_q all come from the search, which ran before the move. What is wrong is value_z: it is the outcome of a game that the stored search never chose to play. Measured over one raw chunk of 10,000,053 records, across plies 16..46 (the only plies where these records occur):

A record standing 186 cp ahead that loses 61% of the time is not noise that averages out — the bias lands on exactly the record where the deviation happened. Trained on, it teaches that winning positions lose; and because value_z is also what the sign metric is scored against, it corrupts the reading before it corrupts the weights.

The record goes rather than just its label because value_z has no absent form: manaka_core::stream::validate requires −1, 0 or +1, unlike cp and visits, which have NO_CP and NO_VISITS. Recovering these positions would mean adding a sentinel to the format, not loosening the rule.

One measured caveat. The upstream card describes the deviations as 5 random opening moves between plies 16 and 32, but the rejected records run from ply 16 to ply 46, with a doubled rate at plies 30–32 (19.11% / 18.99% / 18.33% against 10.93% at ply 16 and 7.43% at ply 46). Why they reach past ply 32 was not established, and no explanation is offered here rather than a guessed one.

The 512-ply cut

The stream manaka-teacher wrote ran to ply 653, against a generator whose stated maximum game length is 320. This repository is that stream with every record above `ply` 512 removed, every game that reached the cap stored as a draw, and the games the generator never finished taken out — 106,568 records dropped at the cap, 516,611 records across 1,024 games relabelled, and 124,507 records across 329 games removed.

Where the long games came from

attic-gensfen enforces --max-ply against the plies played in the current play_game call, not against the move number on the board:

let mut ply_count = 0u16;
// ...
if ply_count >= options.max_ply {
    break (0, "max_ply");
}

The recovery pass (--recover-from) calls play_game again on a position restored out of the existing corpus — its move number already advanced — and ply_count restarts at zero, so a resumed game receives a second full 320-ply budget on top of what is already on the board. A game opening at ply s records s … s+319 on the first pass, resumes at s+319, and records on to s+638. These games open at ply 1 and ply 15, which puts the two terminal plies at 639 and 653.

Measured over the uncut stream, by the ply a game ends on:

break (0, "max_ply") writes result 0, so a game stopped by the cap is already a draw in the corpus, and at 639 and 653 that is very nearly every game ending there. The absence of any such spike at 320 and 334 is the other half of the evidence: the first pass's cap never ends a game in the released stream, because the recovery pass resumed every game, including the ones with no gap to recover. Draws also appear well below 639, which is the same bug seen from another angle and is dealt with below.

This is a bug in the generator, not in the conversion, and it is not fixed here — attic-gensfen is a separate project. Only its effect on the published records is.

What the cut does, and what it costs

• —Records above ply 512 are dropped. A game's plies ascend, so what goes is always a suffix of it; no game loses a middle.
• —Every record of a game that was still being played at ply 512 gets value_z 0. The corpus no longer holds that game's ending, and for 731 of the 1,024 the generator had none either — it stopped them at its own cap and wrote the draw itself. A win label on a game whose end is not in the file states an outcome the data does not hold. value_z has no absent form — manaka_core::stream::validate requires exactly −1, 0 or +1 — so "unknown" is spelt as the draw, which is what the generator does too.

The cost is a ply-dependent label bias, and it is worth being explicit about it: every game alive at `ply` 512 is drawn, by construction, so a record at that ply is always a draw. Reading the drawn share as a function of ply measures this cut and nothing else. If long games matter to what you are training, drop the games holding a record at ply 512 rather than trust their labels — there are 1,024 of them in 14,101,066.

The games the generator gave up on

The uncut stream holds draws well below 639, on 161 distinct end plies, at a rate that climbs the longer the game ran:

Fourteen million games end by ply 331 and not one of them is a draw, so these are not draws in any sense a shogi player would recognise. They are the same max_ply zeros as the cliffs, seen from the other side. The first pass ran with an eval-limit and did not write records whose root score left the bound, so a game's records can stop long before the game did. The recovery pass replayed each such game from its last record with resignation switched off so that it would run to a mate; where it failed to reach one it hit its own cap, wrote that zero over the prefix it had kept, and left the records where they were. What remains is a game labelled "no result was established" whose last record sits between ply 332 and ply 511. (The floor at 332 sits just above the replay's own 320-ply budget. Whether that is the whole explanation was not established.)

329 such games are removed here, taking 124,507 records with them — 0.009% of the stream. They are told apart from the games this repository truncated by the cap itself: a game the cut stopped has a record at ply 512, a game the generator abandoned stops earlier. Whole games go rather than whole records, because the label belongs to the game and not to any one position in it.

Removing them is what makes the earlier sentence true — that a zero in this corpus means the cut and nothing else. Restoring them properly would mean replaying those games again with the eval-limit off and a larger cap, which is work for attic-gensfen and not for a converter; against 14,101,066 games it was not judged worth the wait.

Duplicates

Nothing has been folded. That fold is qleap/Knowledge_distilled_dataset_by_NAGISA_V4, where the uncut 1,372,843,225 records become 1,221,920,925 rows — 11.0% of them share a board with another. The fold has to see every record, because that is where the per-position weight is counted and the candidate scores are averaged, which is why this repository exists rather than being derivable from the folded one.

That parquet was folded before the cut, so it still carries the records above ply 512 and the results this stream redrew. Re-folding these parts would not reproduce it.

Generation

This section is the only source for the settings below. The stream header carries the conventions the converter applied; it has no field for the conditions the games were played under. These are the raw corpus's own, restated from qleap/Raw_dataset_by_NAGISA_V4:

The raw corpus was produced in two passes: an initial run whose eval-limit dropped some records, then a recovery pass (--recover-from) that kept each game's records up to the first gap and replayed the game from there with resignation switched off, so that the replay would run to a mate. It rewrote the kept records' results to the replayed outcome, which is why almost every game here ends in one. It did not reach a mate in 329 of them; those are the zeros that this repository did not write.

Lineage

qleap/Raw_dataset_by_NAGISA_V4 — PackedSfenValue, 1,419,944,833 records
  └─ gensfen-parquet ──▶ parquet
       └─ manaka-teacher ──▶ MPK1, 1,372,843,225 records
            ├─ 512-ply cut
            │    └─ abandoned games removed ──▶ MPK1, 1,372,612,150 records   ← this repository
            └─ manaka-pack ──▶ parquet, 1,221,920,925 rows
                 = qleap/Knowledge_distilled_dataset_by_NAGISA_V4

Every stage of this chain is published, which is not true of the V3 lineage: there the MPK1 repository is the earliest point on the hub, because the USI corpus it was converted from was never uploaded.