chartreuse-verte/ettin-povtense-17m

ettin-povtense-17m

A 17M-parameter encoder that reads two things off a chat message: the point of view it's written in (first / second / third) and its narrative tense (past / present). Both axes can also come back ambiguous, which is a real answer rather than a failure — more on that below, because it's the part of the design I thought about longest.

It exists to catch POV and tense drift in roleplay. A model opens in third-person past, and forty messages later it has slid into second-person present without anyone asking it to. This classifier notices, and the app asks the main LLM to rewrite. It runs on CPU in single-digit milliseconds, so it can sit on a ~1s tick without anyone noticing it's there.

The short version

Using it

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("chartreuse-verte/ettin-povtense-17m")
model = AutoModelForSequenceClassification.from_pretrained("chartreuse-verte/ettin-povtense-17m")

text = "She deactivates her weapon and steps closer. Her breath fogs in the cold."
probs = model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits.softmax(-1)[0]

# 12 cells, ordered pov-major: first|past, first|present, first|ambiguous, second|past, ...
POV   = ["first", "second", "third", "ambiguous"]
TENSE = ["past", "present", "ambiguous"]
grid  = probs.reshape(4, 3)

print(POV[grid.sum(1).argmax()], TENSE[grid.sum(0).argmax()])   # -> third present

Add the grid up before you read it. The 12 probabilities form a 4×3 table — POV down the side, tense across the top. Sum each row to get the POV, sum each column to get the tense, then take the largest in each.

Do not just find the single biggest cell and read its label off. That gives you a different answer, and a worse one. A cell can win the 12-way race with 0.2 while the row it sits in adds up to 0.8 — the model is confident about the POV and unsure which tense goes with it, and picking one cell throws that away. Summing first keeps it.

No thresholds to tune. The sum, then argmax, is the whole decision rule.

Running it without torch

gguf/povtense-17m-q8_0.gguf (19.7 MB) is the same model for llama.cpp.

onnx/model.onnx (67.6 MB, opset 17) is for onnxruntime. Both axes are dynamic, so you can batch and pass any sequence length:

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok  = AutoTokenizer.from_pretrained("chartreuse-verte/ettin-povtense-17m")
sess = ort.InferenceSession("onnx/model.onnx", providers=["CPUExecutionProvider"])

enc = tok(["She steps closer."], return_tensors="np", padding=True, truncation=True, max_length=256)
logits = sess.run(["logits"], {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0]

It matches the torch model exactly — 1.000 agreement on both axes over 200 gold windows.

Why one softmax and not two heads

Two heads would have been the obvious build, and I didn't take it. POV and tense aren't independent in practice — second-person present is the dominant RP register and third-person past is the dominant prose register, so the pair carries information that two separate heads throw away by construction. The cross-product keeps it. Reading it back as marginals means I still get two clean per-axis answers out, so nothing is lost at the interface, and the cost is 12 logits instead of 7.

ambiguous is an abstain class, and it's the point

This is the design decision I'd most want someone to understand before they use this model, because it's easy to mistake for a dumping ground.

None of the training labels are human. Every row was labelled independently by two different LLMs. Where they agreed, agreement became the label. Where they disagreed, that axis became `ambiguous`. So ambiguous isn't a linguistic category I defined — it's the empirical record of where two competent readers couldn't agree, which turns out to be a very good proxy for text that genuinely doesn't commit. Sound effects. Bare dialogue with no narration. Four words of a sentence fragment. Encyclopedic prose that's grammatically third-present but carries no narrative tense at all.

The practical consequence: when the model says `ambiguous`, don't compare. The downstream rule is that an ambiguous read on an axis means no drift call on that axis. Refusing to answer is much cheaper than a spurious rewrite, and that asymmetry is baked into everything below.

To keep the class honest I mined Wikipedia and newswire specifically to teach it. Without expository text, the labeller's third-person-present cell fills up with encyclopedic register and the model quietly learns prose style instead of tense.

How well it works

Evaluated on 500 held-out windows drawn from real conversations, labelled by a third, stronger model that was used nowhere in training — scoring against your own training labeller measures self-agreement, not accuracy.

Point of view — 0.860 accuracy, 0.837 macro-F1

Tense — 0.848 accuracy, 0.853 macro-F1

The model returns ambiguous on about a third of windows. On the two-thirds where it does commit, accuracy is 0.865 (POV) and 0.877 (tense).

Drift decisions

Sampled anchor/candidate pairs, compared the two predictions, and scored whether the drift/no-drift call was right.

The two errors are not equally bad.

A false drift rewrites text the user was happy with. They see it happen, and they stop trusting the feature. A missed drift does nothing at all — no worse than not shipping the feature.

So the model is built to stay quiet when unsure. The missed-drift rates are what I paid for 3.1% and 0.6% false positives. If you need the opposite trade, this checkpoint is the wrong shape.

Training

Fine-tuned on ~32k labelled windows: 20k roleplay turns, 4k Wikipedia, 4k newswire, 4k synthetic passages generated to fill cells the natural corpus barely covers. Deduped by text.

• —5 epochs, lr 4.5e-5, batch 64, max length 256
• —Class-weighted cross-entropy — inverse cell frequency, clipped to 5× — because the natural distribution is roughly 80% third-person past and the rare cells would otherwise be ignored outright
• —Train/val split by conversation, not by row, so windows from one conversation can't straddle the split and leak
• —Gold set never touched during training

Minutes on a 3090. gold_report.json in this repo is the full unedited eval output.

Where it will let you down

• —English only. Nothing else was trained or tested.
• —Short windows. It reads 1–4 sentences, no prior context, the same unit it sees at inference. Hand it three paragraphs and you're outside its training distribution.
• —Roleplay and chat register. Wikipedia and newswire are in there to anchor ambiguous, not to make this a general-purpose prose classifier. It has not been evaluated on literary fiction, screenplays, or academic writing.
• —`first` is the weakest class at 0.727 F1, on 44 supporting examples. Treat first-person calls with more suspicion than the others.
• —No human labels anywhere, including the eval set. Every number on this page is agreement with a strong LLM, not with a person. I think that's a reasonable proxy at this scale and I'd rather say so plainly than imply a rigour that isn't there. A systematically wrong labeller prompt would sail straight through every metric above.
• —It classifies grammatical form. It does not judge whether the writing is any good, and shouldn't be used as if it did.

Training data

The published subset is at `chartreuse-verte/povtense-data` — the Wikipedia, newswire and synthetic portions. The 20k roleplay turns are not published: they're real conversation text, and that isn't mine to open. The dataset card is explicit about what this means for reproduction.

License

MIT, same as the base model. Do what you like with it.

Built for a roleplay chat app. The pipeline that produced it — miners, the two-labeller merge, training, eval, GGUF export — is a near-clone of an earlier emotion-classifier repo of mine with a different label schema.