Modelpublic

nazdef/gpt2medium-en-it-nanochat-gpt2preln-lr2e4-bs2-ga48-wsd-anchor20k-final2e5-webwiki-step13500

sourceHugging Facecc-by-sa-4.0updated 3mo agoView on Hugging Face

0likes17downloads

Model Card

GPT2Medium EN/IT NanoChat GPT2PreLN `step_13500`

This repository publishes the best scalar checkpoint from the medium GPT2PreLN anchor20k run evaluated in the full repo-native sweep currently recorded in this workspace.

This is an ordinary checkpoint release, not the definitive public-facing family base release.

Released checkpoint summary:

—released checkpoint: step_13500.pt
—parent run: stable-recipe-gpt2medium-gpt2preln-k20-wsd-lr2e-4-anchor20k-final2e5-webwiki
—languages: English + Italian
—context window: 2500 tokens
—architecture: GPT-2-style decoder with pre-layernorm blocks
—architecture config: architecture: gpt2, block_type: gpt2_prelayernorm
—training-time parameter count: 337,671,424
—published Transformers-export parameter count: 337,639,424
—hardware class: single consumer GPU (RTX 4060 Ti 16GB) for training, CPU benchmarked selection

This is a base model, not an instruction-tuned chat model.

Why This Checkpoint Was Chosen

The full checkpoint sweep across the evaluated medium run selected `step_13500` as the best checkpoint on the main benchmark metric:

—step_13500: val_loss_mixed = 4.4652
—step_14500: val_loss_mixed = 4.5796
—step_9500: val_loss_mixed = 4.5982
—step_12500: val_loss_mixed = 4.6220
—step_13000: val_loss_mixed = 4.6226

So this release is not “latest wins because timestamp”. It is the current benchmark-selected medium winner for this run family.

Best-Behavior Shortlist

The scalar winner is step_13500, but the behavior shortlist is:

—step_13500
—step_14000
—step_14500

Why step_14000 remained interesting:

—loop_rate = 0.250
—distinct_2 = 0.6081
—repeated_4gram_rate = 0.725
—language_consistency_it = 0.875

Why step_14500 is a good compromise:

—val_loss_mixed = 4.5796
—distinct_2 = 0.5694
—repeated_4gram_rate = 0.750
—language_consistency_it = 0.850

Practical reading:

—step_13500 is the best scalar checkpoint
—step_14000 is one of the cleanest checkpoints on repetition/behavior proxies
—step_14500 trades a little scalar loss for a still-healthy qualitative profile

Main Metrics for `step_13500`

—val_loss_mixed = 4.4652
—val_loss_en = 4.3219
—val_loss_it = 3.5930
—ppl_mixed = 86.9371
—ppl_en = 75.3304
—ppl_it = 36.3446

Behavior snapshot:

—loop_rate = 0.325
—distinct_2 = 0.5268
—repeated_4gram_rate = 0.800
—language_consistency_en = 0.975
—language_consistency_it = 0.850
—cloze_en_contains = 0.16
—cloze_it_contains = 0.30

Source losses:

—books_en = 4.4590
—books_it = 4.5447
—code = 7.2659
—web_en = 5.9069
—web_it = 5.4595
—wiki_en = 3.0141
—wiki_it = 3.0585

Honest short read:

—this is the best scalar medium checkpoint seen so far in this run family
—it is healthier than the later collapsed tail
—it is still a pretrained checkpoint with visible repetition artifacts, not a magically polished model

Training Data

This model was trained on the bilingual EN/IT web + wiki dataset:

—dataset id on disk: 202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
—context window during training: 2500 tokens
—packing length: 2500
—mixing strategy: source_balanced
—validation ratio: 0.05

Main source groups:

—English FineWeb-HQ (epfml/FineWeb-HQ)
—Italian FineWeb2-HQ (epfml/FineWeb2-HQ)
—English Wiki40B (google/wiki40b)
—Italian Wiki40B (google/wiki40b)

How Many Tokens This Checkpoint Saw

Training math:

—sequence length: 2500
—batch size: 2
—grad accumulation: 48
—tokens per optimizer step: 239,904

So step_13500 saw approximately:

—`3.238704B` tokens total
—`K = 9.5913` tokens per parameter relative to the native training-time parameter count

Included Files

This release bundle includes:

—step_13500.pt
—step_13500.safetensors
—model.safetensors
—config.json
—tokenizer files
—training_config.yaml
—run telemetry (best_validation.json, metrics.jsonl, eval_metrics.jsonl, probe_generations.jsonl)
—benchmark bundle (eval_summary.json, comparison.json, benchmark_report.md, benchmark_metrics.json, benchmark_scores.json, benchmark_source_losses.json)

Quick Start

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo_id = "nazdef/gpt2medium-en-it-nanochat-gpt2preln-lr2e4-bs2-ga48-wsd-anchor20k-final2e5-webwiki-step13500"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)

outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    do_sample=True,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

This release is published with `CC-BY-SA-4.0` as the practical downstream posture for the mixed training corpus used here.

The training mix includes:

—FineWeb-HQ / FineWeb2-HQ web data
—Wiki40B English and Italian slices

Downstream users are responsible for checking whether their use, redistribution, or derivative packaging remains compatible with the obligations of the upstream datasets and their terms.

GPT2Medium EN/IT NanoChat GPT2PreLN step_13500