MagicNoThief/handy-editor-lfm2.5-350m

Handy Editor 350M

A 350M model that turns a raw dictated transcript into the text the speaker meant to write.

in : um so the meeting is uh moved to friday no wait thursday at three
out: The meeting is Thursday at three.

Three jobs that in speech are one job: drop filler words, repair punctuation and capitalisation, and — the part models of this size usually get wrong — when the speaker changes their mind mid-sentence, delete the wording they abandoned and keep only what they settled on.

It runs in ~100 ms in 229 MB and scores 68/68 on a self-correction suite — the same score a 4B general-purpose model needs 350 ms and 1.67 GB to reach. 7× smaller and ~3× faster at no cost in accuracy: this exists to run locally, on a laptop with no GPU, without adding a pause you can feel before your text appears.

Fine-tuned from `LiquidAI/LFM2.5-350M` for Handier's on-device enhancement layer.

Prompt format

Chat format, with the transcript as the user turn and no instruction. The task is in the weights; adding the instruction back measured worse (57/68 → 54/68 on an earlier checkpoint) because the model started copying the prompt's own rules into its output.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MagicNoThief/handy-editor-lfm2.5-350m"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "um so the meeting is uh moved to friday no wait thursday at three"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
# Editing is near-deterministic. Sampling turns a working transcript into a
# creative one, which is the one failure users cannot forgive.
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

With llama.cpp, send an empty system message

Not an absent one — the two are not the same thing here, and the difference is worth 7 points.

llama.cpp does not evaluate the GGUF's jinja template; it matches it to a built-in family and renders that. Its chatml renderer emits the system block even when the content is empty, which the jinja template does not:

<|im_start|>system
<|im_end|>
<|im_start|>user
um so the meeting is uh moved to friday no wait thursday at three<|im_end|>
<|im_start|>assistant

In the 59/68 run the model began answering SAME and CHANGED to editing requests — it had stopped recognising the shape of its own input.

With transformers this does not arise: apply_chat_template uses the jinja source, which drops an empty system block, so the plain user-only message list in the snippet above is already correct.

Evaluation

Two independent measurements. Both matter, and a model can pass one while failing the other.

Generation is greedy, so the scores are reproducible rather than a good sample. The latencies are not: across repeated runs the same build measured 94-126 ms depending on what else the machine was doing. Treat them as an order of magnitude and measure on your own hardware if it matters.

`keep` is the number to watch. A model that deletes eagerly scores well on a single total while destroying sentences that were already correct, and that is the failure a user actually notices. 22/22 means it never touched a sentence that did not need touching.

The held-out set is held out by source, not by slicing the training file: real utterances come from a split the training build never reads, synthetic rows use a different seed with every training input excluded by hand, and the builder asserts zero overlap. The suite is 68 hand-written cases, independent of both.

Reproduce with `scripts/enhance-eval/`:

python bench.py --model handy-editor-350m-Q4_K_M.gguf --no-switch --label mine
python eval_heldout.py --model handy-editor-350m-Q4_K_M.gguf \
    --eval handy_eval.jsonl --label mine

Why this checkpoint

Four fine-tunes were compared on the same held-out set:

The two runs from the base checkpoint got worse between step 3,500 and 5,000, failing by producing the right edit and then continuing ("The conference is in Austin. My mistake is in Vienna. That's not right. Austin. …") until the host's length guard rejected the whole thing. Their prompt format was separately verified correct, so that is a training result rather than a data-pipeline one.

Two variables move at once here — base checkpoint and corpus view — so this table says which artefact to use, not which of the two mattered.

Which file do I want?

Both run at the same speed (~100 ms), so this is purely a memory decision.

The interesting result is that Q8_0 and F16 score identically — 2103/2152 each, not approximately but exactly. Q80 therefore costs nothing in quality against the full-precision weights, and F16 buys only disk. Q4KM is the only one carrying measurable quantisation loss, and it is 6 rows in 2,152: real, but far too small to notice in use. Prefer Q80 if the memory is free, Q4KM if it is not, and do not agonise over it.

Q2_K_L is published nowhere, and you should not make one. It is the reason this section exists:

On the 68-case suite Q2K_L looks merely a little degraded — 91%, a number plenty of people would ship on. On the held-out set it gets \_half the edits wrong. A 68-case suite is structurally unable to see that, which is why any quantisation you make must be run through both evaluations before you trust it. Do not infer quality from the suite alone.

Using it in Handier

Nothing to set up: it is Handier's default editor. Settings → Advanced → Local Enhancement turns the layer on and fetches it, and Settings → Models → Enhancement Models offers both builds — Q8_0 as the default, Q4_K_M for machines counting megabytes. The empty system turn is applied automatically, so there is no prompt setting to get wrong.

Running a GGUF you built yourself is still Your Own Model → Choose a GGUF file…; leave How to prompt this model on Fine-tuned for editing, which is what sends the empty system turn instead of Handier's instruction prompt.

Limitations

• —English only.
• —Opinionated punctuation. Sentence case, full stops added, serial commas absent. It will impose that style on your dictation.
• —It deletes on purpose. Cutting retracted wording is the feature, so its mistakes look like missing words rather than garbled ones. Handy keeps the raw transcript in history for exactly this reason; any host should do the same.
• —Not a general instruction-following model. One task, one format. It will not do anything else usefully, and it has no chat ability worth the name.
• —Short utterances dominate its training. Long-form dictation is ~10% of the corpus, and the two suite failures are both long-range retractions.
• —It cannot judge its own edits. Asked whether an edit preserved meaning it gives a confident, meaningless answer: measured over 400 live edits, that pass caught 0 of 10 bad edits and rejected 1 good one. Do not build a verification step on it.

Training

The corpus is 19% examples that need no edit. That share is load-bearing: trained only on corrections, a model learns that something must always be deleted, and starts eating sentences that were fine.

Licence

Inherits the base model's licence: LFM Open License v1.0 (lfm1.0). The terms are the base model's copy, which license_link points at directly: `LiquidAI/LFM2.5-350M/LICENSE`.

Training data is CC-BY-4.0 and requires attribution to `disfl_qa`, `nyralabs/disfluency_speech_english` and `amaai-lab/DisfluencySpeech`.