Evicka/Hanse2-100M-Base

<h1 align="center">Hanse2-100M Base</h1>

<p align="center"> German–English base language model · 99.1M parameters · 20B pretraining tokens · up to 8K context </p>

[image]

[!IMPORTANT] Hanse2-100M-Base is a base model, not a chat assistant. It has not been instruction-tuned, preference-tuned, or safety-aligned. Use it as a text-completion model or as a starting point for continued pretraining and post-training.

Hanse2-100M-Base is a 99,144,320-parameter decoder-only causal language model trained from scratch on 19,999,752,192 German and English tokens. It uses a compact Llama-style architecture, a custom 32K byte-level BPE tokenizer, grouped-query attention, and a three-stage pretraining curriculum that progressively extends context length from 2K to 8K.

The model was trained on a single AMD Radeon RX 9070 XT 16 GB, and the repository includes the tokenizer-training, pretraining, and inference scripts used for the project.

Highlights

• —99.1M parameters, trained from scratch
• —19.999B pretraining tokens (~201.7 tokens per parameter)
• —German + English, approximately 56% / 44% by planned token share
• —2K → 4K → 8K progressive context curriculum
• —32K custom byte-level BPE tokenizer
• —Single-GPU training on an AMD Radeon RX 9070 XT 16 GB
• —Reproducible project code for tokenizer training, pretraining, and inference
• —Apache-2.0 release

Model overview

Architecture

Pretraining

Pretraining used three consecutive phases while keeping the effective optimizer batch at approximately 131,072 tokens per optimizer step.

[image]

Data mixture

Approximate token share over the full curriculum:

This corresponds to approximately 56% German / 44% English.

Optimization

Final held-out evaluation loss after Phase 3 was approximately 2.825 on the fixed bilingual evaluation split.

Tokenizer

Hanse2 uses a custom 32,000-token byte-level BPE tokenizer trained on a German–English mixture.

Several tokens were reserved for later instruction/tool post-training. They were not used as a chat format during base pretraining, and this checkpoint should not be prompted with an assumed chat template.

Evaluation

The following model-comparison results were produced with the EleutherAI LM Evaluation Harness using the same local setup for Hanse2-100M and Supra-50M:

• —0-shot
• —no chat template
• —bfloat16
• —identical task implementations
• —identical random seeds

Hanse2 remains close to Supra-50M on several English benchmarks while substantially improving the German linguistic evaluation.

Long-context diagnostic

The model is configured for 8,192 tokens and produces finite forward passes at ~7.7K tokens. This does not imply reliable retrieval across the full context window.

A small base-LM continuation test placed a one-token identifier earlier in a synthetic context and measured its next-token rank against nine decoys.

The model can process long inputs, but retrieval degrades when relevant information is several thousand tokens away from the prediction point. Treat this as a small synthetic diagnostic, not a standardized long-context benchmark.

Quickstart

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/Hanse2-100M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "Artificial intelligence is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

[!NOTE] This is a completion model. Prompts such as sentence beginnings, paragraphs, or documents to continue are more appropriate than chat-style system/user/assistant messages.

Reproducibility

The repository includes the core scripts needed to reproduce or adapt the project:

The training pipeline includes exact token-budget accounting, resumable checkpoints, deterministic held-out evaluation, and separate context/batch settings for each phase.

The initial training/tokenizer code was based on the Apache-2.0-licensed Supra-50M training scripts and was substantially rewritten and extended for Hanse2. No Supra model weights were used to initialize Hanse2.

Re-running the scripts does not by itself guarantee bit-identical reproduction across different hardware, driver, PyTorch, ROCm, dataset revisions, or nondeterministic kernels.

Intended use

Hanse2-100M-Base is intended primarily for:

• —research on small language models
• —German–English pretraining experiments
• —bilingual data-mixture studies
• —tokenizer and architecture experiments
• —progressive context-extension experiments
• —benchmarking and ablations
• —continued pretraining
• —supervised fine-tuning / post-training research
• —education and consumer-hardware experimentation

Limitations

At ~99M parameters, Hanse2 is a small research model. Expect:

• —weak factual recall and hallucinations
• —weak arithmetic and multi-step reasoning
• —repetition or topic drift
• —brittle knowledge of specific entities
• —limited long-range coherence and retrieval
• —unstable behavior outside its training distribution

The checkpoint has no instruction tuning, preference tuning, RLHF, refusal training, or production safety guardrails. It may generate inaccurate, biased, offensive, harmful, or otherwise undesirable continuations.

The configured 8K context length should not be interpreted as reliable 8K retrieval capability; see the long-context diagnostic above.

License

Released under the Apache License 2.0. See `LICENSE`.

Acknowledgements

• —Hugging Face — Transformers, Datasets, Tokenizers, FineWeb, FineWeb-Edu, and FineWiki
• —EleutherAI — LM Evaluation Harness
• —SupraLabs — original Apache-2.0 Supra-50M training/tokenizer scripts used as the starting point for the project code

Citation

If you use Hanse2-100M-Base in a project, a link to the model repository is appreciated.

@misc{brauer2026hanse2base100m,
  author = {Erik Brauer},
  title  = {Hanse2-100M-Base},
  year   = {2026},
  url    = {https://huggingface.co/Evicka/Hanse2-100M-Base}
}