qleap/MPK1_dataset_by_NAGISA_V3

MPK1 dataset by NAGISA V3

Shogi teacher positions from self-play of the search engine attic reading the NNUE weights NAGISA_V3, carried in MPK1 — the raw record stream, one record per searched position, before duplicate boards are folded into one. This is the unprocessed input behind qleap/Knowledge_distilled_dataset_by_NAGISA_V3: no folding, no deduplication, no policy normalisation, no shuffling beyond the one the games arrived in. 2,858,094 games, 300,045,733 positions.

qleap/MPK1_dataset_by_NAGISA_V4 is this same format one generation on. qleap/Raw_dataset_by_NAGISA_V4 is a different format altogether — the 60-byte PackedSfenValue stream V4's generator wrote. There is no such stage in this corpus's lineage: V3's generator wrote USI text, manaka-teacher read it and these records are the first binary form it took. So this repository is not the V3 counterpart of that one; it is the counterpart of the V4 MPK1 repository, and the earliest published point in its own chain.

Do not pool it with the V4 stream without reading both headers. They declare different conventions — mate_scale 30000 here against 32000 there, and an effective mate_slack of 320 against 100. The same cp is a mate in one and an ordinary evaluation in the other, and nothing downstream checks.

Files

`load_dataset` does not read this. MPK1 is a binary record stream, not a format datasets knows; the repository is an archive, and the folded parquet at qleap/Knowledge_distilled_dataset_by_NAGISA_V3 is what a trainer loads. Fetch the files and walk them:

from huggingface_hub import snapshot_download

path = snapshot_download("qleap/MPK1_dataset_by_NAGISA_V3", repo_type="dataset")

# one part, decompressed
unzstd -c part-00000.mpk.zst > part-00000.mpk

The parts do not concatenate. Each one is a complete MPK1 stream and opens with its own 24-byte header, so cat part-*.mpk.zst | unzstd puts a header in the middle of the record run and a reader walks off the framing at that byte. Decompress and read them one at a time; the headers are byte-identical, so nothing is lost by dropping all but the first.

A game can straddle a part boundary. The cut is at a record count, not at a game boundary — 28 of the 29 joins land inside a game, and the one that does not is a coincidence rather than a rule. A reader that needs whole games either reads the parts in name order as one logical stream, or discards the partial game at each end.

Format

Defined in manaka_core::stream. Little-endian throughout.

header, 24 bytes
  magic u32 "MPK1" | version u16 | packed_len u16
  pov u8 | policy_source u8 | mate_scale u16
  fv_scale u16 | mate_slack u16 | eval_coef f32 | flags u32

record, variable length — 12 + packed_len + 8·n_cand bytes
  ply u16 | value_z f32 | value_q f32 | n_cand u16
  packed [u8; packed_len]
  candidates × n_cand : mv u16 | visits u32 | cp i16

There is no record count in the header and no index: a part is read by walking it. There is no legal-move list and no legality mask either — a reader that needs one regenerates it from packed.

What this stream declares

Every part opens with the same 24 bytes:

4d 50 4b 31 01 00 60 00 00 01 30 75 1c 00 00 00 89 05 bd 44 00 00 00 00

`mate_slack` reads 0 and that means "not stated", not zero. The field was added to MPK1 after these parts were written, in bytes a version 1 writer left reserved, so a reader falls back to manaka_common::card::DEFAULT_MATE_SLACK, which is 320 — and 320 is this corpus's number. The threshold is mate_scale − slack, so |cp| ≥ 29680 is a mate here. A reader that substitutes some other slack reclassifies real evaluations as mates, or mates as evaluations, and nothing complains.

320 is not a round number picked for looks. The generator substitutes the length of the PV when YaneuraOu reports a mate as an out-of-range score cp, and a long PV runs well past YaneuraOu's MAX_PLY of 246: measured over one shard of 14.3M positions, mates reach ply 315. 320 is the smallest round value that still covers them. Widening it to 512 would newly claim |cp| 29488..29680, which holds roughly 1,300 rows across this corpus — each of which would switch from a softmax over evaluations to a one-hot.

`FV_SCALE` is what makes a centipawn a centipawn. The raw value an NNUE accumulates becomes a score only after division by this number. Read the same weights at the engine default of 16 and every cp here would be 1.75× larger. Before mixing this data with anyone else's, check that this number agrees; teacher data blended across two scales trains on numbers belonging to neither.

What packed holds

96 bytes, manaka_core::pack. The position itself — not an engine's input layer; a feature space can change without the corpus being rebaked.

byte
 0..81   the board, one byte per square, in square-index order
81..88   Black's hand, one count per piece kind
88..95   White's hand, same order
    95   side to move, 0 Black / 1 White

Square 0 is 9a and square 80 is 1i: square = (9 − file) × 9 + (rank − 1), so ranks 1–3 are White's camp and 7–9 are Black's.

A board byte carries the colour in bit 4 (0 Black, 1 White) and the piece kind in the low nibble; an empty square is 0xFF, which cannot collide because no valid byte sets the high bits.

Hands are seven counts each, one byte apiece, and not in the board's kind order — hands run P, L, N, S, G, B, R, the reverse direction. Maxima are 18, 4, 4, 4, 4, 2, 2. This is the one place in the format where the same seven kinds appear under two orderings, so it is worth checking against rather than assuming.

A candidate is a move, not a label

mv is a move16 — chisaki's move representation, verbatim.

 bit  15 │ 14 │ 13 ──────── 7 │ 6 ──────── 0
      ✗  │ pr │ from or 81+kind │     to

to in bits 0–6, the origin square in bits 7–13, promotion in bit 14, bit 15 unused. A drop puts 81 + hand index in the origin field instead of a square, so the dropped kind is readable from the move itself.

A reader needs no move generator, and therefore no shogi library. A policy label is a direction and a destination, so a pawn stepping to 7f and a lance running there are the same number, and which piece was lifted is not recoverable from it. A move16 carries the origin outright, so both the label and the lifted piece fall out of bit arithmetic — seven lines of it in any language.

Candidate lists

MultiPV was 5, and the width of a record's list is its own n_cand, not a number declared beside the file:

1,401,425,684 candidate scores in all, 4.67 to a record. Lists shorter than 5 are not truncation: the generator realigns every candidate's score to a common depth and drops the ones with no settled value at that depth. There is no record with an empty list — one would contribute zero loss and look like learning.

Over all 1,401,425,684 of those scores the extremes are exactly ±29999, and nothing reaches ±30000. That is the mate scale showing itself: mates are stored as ±(30000 − ply) and no ordinary evaluation comes near.

`cp` is a score, not a probability. No softmax has been applied — the temperature and the mate one-hot are applied at the parquet conversion, so this corpus can be re-softmaxed at a different temperature without regenerating it. cand_cp is also not sorted: the order is the MultiPV rank the engine assigned, and the values are the realigned ones, so a rank inversion is normal.

Reading it

import struct

FIXED, CAND = 12, 8

def records(path):
    with open(path, "rb") as f:
        head = f.read(24)
        assert head[:4] == b"MPK1"
        packed_len = struct.unpack_from("<H", head, 6)[0]
        blob = f.read()
    i = 0
    while i < len(blob):
        ply, value_z, value_q, n = struct.unpack_from("<HffH", blob, i)
        packed = blob[i + FIXED : i + FIXED + packed_len]
        base = i + FIXED + packed_len
        cands = [
            struct.unpack_from("<HIh", blob, base + CAND * k)[0:3:2]  # (mv, cp)
            for k in range(n)
        ]
        yield ply, value_z, value_q, packed, cands
        i = base + CAND * n

(unzstd the part first, or wrap the file in zstandard.ZstdDecompressor.)

Row order

Games are shuffled; the plies inside one are not. A game's records are a contiguous run in ascending ply, incrementing by exactly 1 — measured over all 300,045,733 records, zero holes inside a game — and only the order of the games was permuted. So the games are recoverable:

A new game starts at every record whose `ply` does not increase over the previous record's. There is no game id and no delimiter.

ply is the position's own move number, taken from the board rather than counted from the start of the game, and the openings come from books at ply 24 and ply 32 — so a game opens at `ply` 24 or 32 and at no other value, never 0, and the 320-ply cap is on moves played from the book position, which is why ply runs past 320. 54.1% of games open at 24, 45.9% at 32.

Measured over all 300,045,733 records:

(Walking the parts turns up 27 further apparent game starts at scattered plies. Those are the part boundaries that land mid-game, not games; see files.)

A random row split is fine. The label is a search score that depends only on the position, so nothing has to be held out at a game boundary: slice off whatever fraction you want for validation.

Value targets

value_q reaches exactly −1.0 and +1.0, which is the mate signum rather than a saturated tanh.

The upstream USI corpus states its result from Black and its cp from the side to move; the conversion rotated result so that both are the side to move's here, which is what the header's pov declares. A reader that applies the upstream card's rotation a second time inverts the target on half the rows, and the loss curve still looks like training.

Draws are stored as a float zero whose sign bit is sometimes set — -0.0 arrives when a drawn game's record was rotated from White's side. value_z == 0.0 is true for both; a bitwise comparison against 0x00000000 is not. 461,374 of the 920,911 drawn records carry the negative zero.

The root `cp` is not stored, only `value_q`. It can be read back as 1512.173 · atanh(value_q) where the position is not a mate, and cannot where it is: a mate's value_q is the signum, and the distance is gone. The per-candidate cp values are stored unsquashed and are a different measurement — the corpus aligns them at a shallower depth than the root score, so max(cp) over the candidates does not equal the root.

Duplicates

Nothing has been folded. 56,326 opening positions were played 2.86M times, so early positions recur:

• —unique sfen (move number included) upstream: 282,637,740 — 5.8% of rows are repeats
• —folding on the 96 packed bytes alone, which ignores the move number, leaves 280,621,202 rows: 6.47% of rows share a board with another

That fold is qleap/Knowledge_distilled_dataset_by_NAGISA_V3, which is what a trainer reads. The fold has to see every record, because that is where the per-position weight is counted and the candidate scores are averaged — which is why this repository exists rather than being derivable from the folded one.

Generation

This section is the only source for the settings below. The stream header carries the conventions the converter applied; it has no field for the conditions the games were played under.

The engine and the evaluation function are two different things. attic did the searching; NAGISA_V3 is the set of weights it loaded.

Perpetual check is scored as an ordinary repetition draw rather than a loss for the checking side: the generator had no board to detect it with. A known approximation, and the reason a small share of value_z zeros are not really draws.

Conversion

manaka-teacher read the upstream USI corpus row by row and wrote these records. Per row it re-parsed the sfen, regenerated the legal moves, and required that every candidate the engine reported be legal on the board it decoded and that the played move be one of the candidates — a check on the sfen and the rotation, not on the engine. 300,045,733 rows in, 300,045,733 records out, 0 rejected.

The played move itself does not reach the stream. It was the corpus's mate one-hot target, and the one-hot is re-derived at the parquet conversion from the leading candidate.

Lineage

USI corpus (sfen strings, USI moves, parquet)     ← not published
  └─ manaka-teacher ──▶ MPK1, 300,045,733 records   ← this repository
       └─ manaka-pack ──▶ parquet, 280,621,202 rows
            = qleap/Knowledge_distilled_dataset_by_NAGISA_V3

The USI corpus is not on the hub. Every revision of qleap/Knowledge_distilled_dataset_by_NAGISA_V3 holds the folded parquet — there is no commit to check out that reaches the stage above this one. So this repository is the earliest published point in the chain, and the generation table above is the only account of what produced it.