BytesTalk/PersonaMini-1-small

PersonaMini-1 small

A 28.8M-parameter GPT-style roleplay & chat model, trained entirely from scratch by BytesTalk — no existing model was fine-tuned; every weight was learned from random initialization. It punches well above its weight on conversational behavior for its size, while staying honest about the hard limits of a model this small (see Limitations).

Mature content. This is an uncensored roleplay model and can produce adult/NSFW text. Intended for 18+ use only.

Capabilities

• —Coherent, in-character short replies; holds a given persona
• —Answers relevantly instead of dumping its identity
• —Empathetic responses; acknowledges the user; short-context memory recall
• —Roleplay and character chat — its core purpose

Limitations

• —World knowledge is weak. It reliably knows only a small set of everyday concepts; for anything obscure it will guess. This is a hard parameter-count ceiling, not a fixable bug.
• —Two-name role tracking ("I'm the character / you're the user") can still wobble.
• —Occasional incoherence or invented details — reduced through training, not eliminated.
• —Best for short roleplay / chat, not facts, math, code, or long documents.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("bytestalkai/PersonaMini-1-small")
model = AutoModelForCausalLM.from_pretrained("bytestalkai/PersonaMini-1-small").eval()

msgs = [{"role": "user", "content": "You are Mia, a flirty bartender. Hi Mia!"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, do_sample=True, temperature=0.6,
                     top_p=0.9, top_k=40, repetition_penalty=1.2,
                     eos_token_id=50256, pad_token_id=50256)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Recommended sampling: temperature≈0.5–0.6, top_p≈0.9, top_k≈40, repetition_penalty≈1.2. Lower temperature reduces invented details.

Chat format

### USER:
{optional persona line}
{your message}

### ASSISTANT:
{reply}<|endoftext|>

An optional persona/system line goes at the top of the user turn, e.g. You are Luna, a shy witch.

Architecture

GPT-2-compatible decoder, exported to standard GPT2LMHeadModel (the safetensors here match the original checkpoint to a max logit difference of ~2e-5):

Training pipeline

The model was built with a modern small-model recipe: pretrain → staged SFT → iterative distillation-by-repair. A larger instruction-following LLM was used only during data preparation as an automated judge and reviser — it is not shipped, embedded, or queried at inference time.

1. Pretraining

Trained from random initialization on a mixed corpus (~0.4B tokens) of web/educational text, short stories, and roleplay/dialogue data, so the base learns general English plus the target domain. Best-validation checkpointing was used as an early-stopping guard.

2. Supervised fine-tuning (staged, to avoid one common failure)

Instruction/roleplay SFT was split into stages because mixing everything into one blob made earlier versions bland and caused identity to "bleed" into unrelated answers:

• —Stage A — rich SFT (no identity): roleplay + chat + Q&A only, to teach format, turn-taking, and immersion while preserving the base's richness.
• —Stage B — light identity pass with replay: a small pass adds the model's self-identity and a few grounded definitions. It is mixed with a replay sample of Stage-A data to prevent catastrophic forgetting — without replay, an identity-only pass made the model answer every prompt with its self-introduction.

3. Iterative distillation by repair (the main quality driver)

For a bank of prompts (roleplay with diverse personas, chat, empathy, memory, identity, and persona-grounded perspective scenarios), the pipeline:

1. 1.generates several candidate replies from the current model,
2. 2.has the judge model pick the best/worst, score the best (1–5), and — when the best is weak — rewrite it into an exemplary short reply,
3. 3.uses those rewrites (plus the model's own high-scoring replies) as supervised targets (text-level distillation / rejection-sampling fine-tuning).

This was run for two rounds, each time using the improved model as the generator. The model measurably improved between rounds — average self-reply quality rose 2.55 → 3.25 / 5, and the fraction of replies needing a rewrite fell 89% → 62% — evidence the loop was closing the gap.

Measures taken (methodology notes)

• —Checkpoint selection. On hard, free-form data, per-token validation loss barely beats the base even while behavior improves, so we save the final (fully-trained) checkpoint rather than the lowest-val one, which would otherwise return an essentially untrained model.
• —Anti-forgetting replay. Every light continuation pass mixes in a replay sample of prior data, so new skills are added without erasing old ones.
• —Judge design. The automated judge used a strict, priority-ordered rubric: relevance first (a reply that ignores the question — e.g. self-introducing instead of answering — is scored as a non-answer and rewritten), then perspective (the character is the assistant; never call the user by the character's name; recall the user's own stated name), no hallucination (never invent names/facts the user didn't provide), persona-detail use, and no repetition. Candidate order was randomized to remove position bias, and each chosen reply got an absolute quality score so only genuinely-good replies became training targets.
• —Repetition handling. Replies that echo an earlier turn are penalized/rejected explicitly.
• —Preference optimization tried and dropped. A DPO pass was implemented and tested, but on these teacher-vs-model pairs the quality gap was so large that DPO over-optimized (reward margin exploded, outputs degraded) regardless of the KL weight. Supervised distillation gave the real gains, so the released model is the distillation checkpoint, without DPO.
• —Export verification. The safetensors and GGUF exports were checked to reproduce the original model's outputs before release (logit match ~2e-5).

Evaluation

Evaluation was behavioral, targeting the specific failure modes above: relevance, empathy, name-acknowledgement, in-character roleplay, identity, short-context memory, perspective/role tracking, and hallucination — compared side-by-side across successive versions.

Intended use & out-of-scope

Intended for short-form roleplay and casual chat by adults. Out of scope: factual question answering, reasoning/math/code, professional or safety-critical use, and any use by minors or that violates applicable law.

License

MIT. Use responsibly. 18+ only.

Family

• —PersonaMini-1-small — 28.8M (this model)
• —PersonaMini-1-medium — 63.2M
• —PersonaMini-1-big — 160M