iamshnoo/asia_with_metadata_500m

asiawithmetadata_500m

Summary

This repo contains the asia 500m model at the final 10k-step checkpoint for the metadata localization project. It was trained from scratch on the project corpus, using the Llama 3.2 tokenizer and vocabulary.

Variant Metadata

• —Stage: pretrain
• —Family: local_continent
• —Size: 500m
• —Metadata condition: with_metadata
• —Base model lineage: Trained from scratch; tokenizer/vocabulary from meta-llama/Llama-3.2-1B

Weights & Biases Provenance

• —Run name: 11/10/2025_00:37:00_asia_with_metadata_500m
• —Internal run URL: https://wandb.ai/iamshnoo/nanotron/runs/s353qw74
• —Note: the Weights & Biases workspace is private; public readers should use the summarized metrics and configuration below.
• —State: finished
• —Runtime: 75h 30m 45s

Run Summary

• —KPI/train_lm_loss: 2.0426
• —KPI/train_perplexity: 7.7107
• —KPI/val_loss: 2.1908
• —KPI/val_perplexity: 8.942
• —KPI/consumed_tokens/train: 41,943,040,000
• —_step: 10,000

Training Configuration

• —train_steps: 10,000
• —sequence_length: 2,048
• —micro_batch_size: 8
• —batch_accumulation_per_replica: 64
• —learning_rate: 0.003
• —min_decay_lr: 0.0003
• —checkpoint_interval: 1,000

Training Curves

Static plots below were exported from the private Weights & Biases run and embedded here for public access.

Train Loss

[image]

Validation Perplexity

[image]

Throughput

[image]

Project Context

This model is part of the metadata localization release. Related checkpoints and variants are grouped in the public Hugging Face collection Metadata Conditioned LLMs.

• —Training data source: News on the Web (NOW) Corpus
• —Project repository: https://github.com/iamshnoo/metadata_localization
• —Paper: https://arxiv.org/abs/2601.15236

Last synced: 2026-04-02 14:41:08 UTC