violetxi/hparam-92m-block-mtp-truncated-peak_lr-1e-3-wu-0p01-s42

92M Block-MTP — step 237865

Condition: 92m-blktrunc-lr1e3-wu01-s42. This branch contains ready-to-load model weights, config, tokenizer and the exact shared-core implementation. Each checkpoint has its own step-N branch. Install the packages in requirements.txt.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "violetxi/hparam-92m-block-mtp-truncated-peak_lr-1e-3-wu-0p01-s42"
revision = "step-237865"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda" if torch.cuda.is_available() else "cpu").eval()
inputs = tokenizer("Pe2e4 Pe7e5 Ng1f3", return_tensors="pt").to(model.device)
generated = model.generate(**inputs, max_new_tokens=20, do_sample=True,
                           temperature=1.0, top_k=0, top_p=1.0)
print(tokenizer.decode(generated[0], skip_special_tokens=True))

Evaluation semantics

Three recurrent passes score three fixed, causally ordered query slots. generate() commits that three-token block before the next recurrent block; the mask token (ID 81) cannot be generated. Sampling and greedy generation, padding and num_return_sequences are supported; KV caching and beam search are not supported. Context is 1,024 logical tokens including the query block.

model.block_logits(input_ids) returns [batch, 1, 3, vocab] for the next block. model(input_ids, labels=input_ids) uses the registered dense non-overlapping block loss: logits[:, t] scores token t+1, with every group of three seeing only the prefix before that group. It is not ordinary teacher-forced NTP. Benchmark inference is not performed by publication. Condition selection remains governed by the sweep's validation and collapse rules.

Safetensors values preserve the native model tensors exactly. Optimizer and RNG recovery state remains in the native archive; it is not needed for HF inference. training_summary.json binds this export to the native checksum and source versions. Adapted core code licenses are included alongside it.