achille-fusco/gpt2_ParFindFast_eng_nl_zho

BabyLM 2026 - Multilingual GPT-2 (ParadigmFinder)

Track: BabyLM 2026 Multilingual Architecture: GPT-2 Tokenizer: ParadigmFinder with boundary discovery support

A multilingual GPT-2 language model trained for the BabyLM 2026 multilingual setting on English, Dutch, and Chinese. This release is intended as the ParadigmFinder counterpart to the multilingual MoP baseline: it uses the same model family and the same multilingual_selection training data for language-model training, while replacing the tokenizer with a ParadigmFinder-based segmentation pipeline.

The tokenizer and model were exported together and can be loaded directly from the Hub. Since the tokenizer implementation is custom, loading requires trust_remote_code=True.

Model Details

Architecture

Tokenizer

The model uses a custom ParadigmFinder tokenizer exported as EnhancedParadigmTokenizerWrapper. In this multilingual configuration, tokenizer training uses a soft per-language vocabulary budget and boundary discovery support designed to remain compatible with space-free scripts such as Chinese.

Tokenizer highlights:

• —Boundary discovery enabled: true
• —Boundary discovery mode: space_free_only
• —Shorter-first sentence ordering: true
• —Explicit word-boundary insertion during tokenization: disabled (use_word_boundaries=false)
• —Vocabulary budget: 16,384 token slots per language
• —Languages: English (eng), Dutch (nld), Chinese (zho)

Special tokens:

• —<pad> = 0
• —<unk> = 1
• —<s> = 2
• —</s> = 3

This repo includes the tokenizer code and supporting files needed for loading:

• —tokenizer.py
• —boundary_discovery.py
• —preprocessing.py
• —paradigm_utils.py
• —paradigms.json
• —multilingual_meta.json

Training Data

This model was trained on the BabyLM multilingual data selection stored locally as multilingual_selection, with the three languages:

• —English (eng)
• —Dutch (nld)
• —Chinese (zho)

For comparability with the multilingual MoP baseline, the GPT-2 model training uses the same multilingual corpus selection. In this experiment line, tokenizer training and model training were aligned to the same multilingual data source.

Training Procedure

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "achille-fusco/gpt2_ParFindFast_eng_nl_zho"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

prompt_en = "The child looked at"
inputs = tokenizer(prompt_en, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_nl = "Het kind keek naar"
inputs = tokenizer(prompt_nl, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_zh = "孩子看着"
inputs = tokenizer(prompt_zh, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

If the environment has a read-only Hugging Face cache, set a writable cache first:

export HF_HOME=/tmp/hf_cache

Evaluation

This model was evaluated with the BabyLM 2026 multilingual evaluation pipeline across zero-shot and fine-tuned tasks.

Summary scores:

Selected zero-shot results:

These numbers correspond to the collated submission artifact produced for this export.

Limitations

• —This is a small multilingual BabyLM-scale model and is not intended as a production general-purpose LM.
• —The tokenizer relies on custom remote code, so some environments may require extra care when loading.
• —No explicit language identifier is used; mixed-language prompting may lead to unstable behavior.
• —Compared with the multilingual MoP baseline, this model is stronger on some Chinese and compositional zero-shot evaluations, but weaker overall after finetuning.

Citation

If you use this model, please cite the model repository and the BabyLM shared task.

@misc{fusco2026parfindfast,
  author       = {Fusco, Achille},
  title        = {{BabyLM 2026 Multilingual GPT-2 (ParadigmFinder)}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/achille-fusco/gpt2_ParFindFast_eng_nl_zho}},
  note         = {BabyLM 2026 multilingual model release with ParadigmFinder tokenizer.}
}