NetherQuartz/ilo-toki-1.1-MiLMMT-46-1b-merged

ilo toki 1.1 — MiLMMT-46 1B

A translator between Toki Pona and English, Russian and Vietnamese. Small enough to run on a phone: it powers ilo toki, which does all of its translation on device.

This repository holds both the merged weights and GGUF builds, so there is one place to look rather than a repository per format.

Version 1.1 replaces `ilo-toki-MiLMMT-46-1b-merged`. See What changed in 1.1 for what got better and what got worse — it is an improvement on balance, not on every axis.

Prompt format

The model keeps the prompt format of its base, and there is no chat template — do not wrap the input in one.

Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:

The translation follows the final <target language>: line and ends at the model's end-of-generation token. Either side can be the source:

Translate this from Russian to Toki Pona:
Russian: Я тебя люблю.
Toki Pona:

Language names are written out in full — Toki Pona, English, Russian, Vietnamese. Getting the format wrong does not fail loudly: the model keeps producing fluent text while silently ignoring the requested target language.

Toki Pona is written in lower case; capitalization in the input is not something the model expects. Terminal punctuation is optional — unlike in 1.0, adding or dropping a final . or ? no longer changes the answer much.

Which file to use

The quantizations sit unusually close together because the 262k-token embedding matrix is about a third of the model and quantizes the same way in all of them. Q80 therefore costs only 0.05 GB more than Q6K and 0.35 GB more than Q4KM, which is why the app ships it: on a phone the difference between these files is small, while the difference between fitting in RAM and not is enormous.

Running it

With llama.cpp:

llama-completion -m ilo-toki-1.1-MiLMMT-46-1b-Q8_0.gguf --temp 0 --top-k 1 \
  -p "Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:"

Greedy decoding is what this model is meant to be run with. There is one right answer per input, and sampling only ever walks away from it.

With transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NetherQuartz/ilo-toki-1.1-MiLMMT-46-1b-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Translate this from Toki Pona to English:\nToki Pona: jan li moku e kili\nEnglish:"
inputs = tokenizer(prompt, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=64, do_sample=False)[0]))

How it was built

A LoRA adapter (`NetherQuartz/ilo-toki-1.1-MiLMMT-46-1b`) trained with TRL SFT — rank 64, targeting the attention and MLP projections, plus 15 764 individual embedding rows through PEFT's trainable_token_indices — merged into MiLMMT-46-1B-v0.1 and quantized with llama.cpp.

The base is a 46-language translation model from Xiaomi Research, so the fine-tune starts from a model that already translates rather than from a general purpose one.

Training data

Alongside tok↔x pairs the mix includes x↔y pairs between the natural languages, meant to keep their generation fluent.

What changed in 1.1

• —No Minecraft corpus. 1.0 had one, and jan — which appears in a large share of all Toki Pona sentences — came back as Player: jan li tawa ma gave «Player moves», jan li pali e tomo gave «Building a Structure». Gone in 1.1: «Someone went outside», «Someone built the house».
• —Terminal punctuation dropped with p = 0.25 during training. 1.0 gave visibly different answers with and without a final .; 1.1 is stable, and where it does differ it is only wording.
• —Fewer dropped clauses. 1.0 turned soweli lili li lape lon tomo into «Rabbit sleeps», losing the location; 1.1 keeps it.

Known limitations

Measured against 1.0 on 95 prompts across the three languages, both directions. 1.1 wins on the above, and loses on these — all worth knowing before relying on it:

• —`la` is often read as a conditional. mi wile lape la mi tawa tomo gives «If I want to sleep then I go home» where the particle here is contextual, not an if. Sometimes the second clause is mangled with it.
• —`sona e toki pona` can name the wrong language. mi sona e toki pona gives «I know Russian». Other constructions around toki pona are fine.
• —Invented specifics. Short inputs sometimes get concrete detail that is not in them — a place name, an extra adjective, an extra clause.

• —Unmarked features get a fixed default rather than a contextual reading. Toki Pona marks neither number nor tense, and la does not say which relation it joins two clauses with. Every one of those is inferred, and 1.1 infers the same way almost every time: bare mi comes back as «we» in six of seven sentences where 1.0 said «I», an unmarked verb tends to come back past, and la tends to come back conditional. None of these is wrong — mi covers «we» and mi mute is optional — but a translator that answers mi tawa tomo with «We went home» is picking the less expected of two valid readings, consistently.

A note for anyone re-merging this adapter

gemma3 ties lm_head to embed_tokens, and this adapter trains individual embedding rows (trainable_token_indices) with ensure_weight_tying: false, so the tied output head read the base embeddings throughout training. A plain merge_and_unload() writes the deltas into the shared tensor and the head suddenly sees an update it never saw — which produces a model that repeats a single token forever.

The weights here were merged with the output head untied and left at the original embeddings, which reproduces training exactly. That is also why model.safetensors carries a separate lm_head.weight and tie_word_embeddings is false. The 1.0 adapter put a full LoRA on embed_tokens instead; the trap, and the fix, are the same either way.

Licence

Gemma Terms of Use, inherited through the base model.