BytesTalk/PersonaMini-1-small

1---2license: mit3language:4- en5library_name: transformers6pipeline_tag: text-generation7tags:8- roleplay9- conversational10- gpt211- tiny12- from-scratch13- distillation14- not-for-all-audiences15---16 17# PersonaMini-1 small18 19A **28.8M-parameter** GPT-style roleplay & chat model, **trained entirely from scratch** by20**BytesTalk** — no existing model was fine-tuned; every weight was learned from random21initialization. It punches well above its weight on conversational *behavior* for its size, while22staying honest about the hard limits of a model this small (see **Limitations**).23 24> **Mature content.** This is an uncensored roleplay model and can produce adult/NSFW text.25> Intended for **18+** use only.26 27---28 29## Capabilities30- Coherent, in-character short replies; holds a given persona31- Answers *relevantly* instead of dumping its identity32- Empathetic responses; acknowledges the user; short-context memory recall33- Roleplay and character chat — its core purpose34 35## Limitations 36- **World knowledge is weak.** It reliably knows only a small set of everyday concepts; for37  anything obscure it will guess. This is a hard parameter-count ceiling, not a fixable bug.38- **Two-name role tracking** ("I'm the character / you're the user") can still wobble.39- **Occasional incoherence or invented details** — reduced through training, not eliminated.40- Best for **short roleplay / chat**, not facts, math, code, or long documents.41 42---43 44## Usage45 46```python47from transformers import AutoModelForCausalLM, AutoTokenizer48import torch49 50tok = AutoTokenizer.from_pretrained("bytestalkai/PersonaMini-1-small")51model = AutoModelForCausalLM.from_pretrained("bytestalkai/PersonaMini-1-small").eval()52 53msgs = [{"role": "user", "content": "You are Mia, a flirty bartender. Hi Mia!"}]54prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)55ids = tok(prompt, return_tensors="pt").input_ids56out = model.generate(ids, max_new_tokens=80, do_sample=True, temperature=0.6,57                     top_p=0.9, top_k=40, repetition_penalty=1.2,58                     eos_token_id=50256, pad_token_id=50256)59print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))60```61 62**Recommended sampling:** `temperature≈0.5–0.6`, `top_p≈0.9`, `top_k≈40`, `repetition_penalty≈1.2`.63Lower temperature reduces invented details.64 65### Chat format66```67### USER:68{optional persona line}69{your message}70 71### ASSISTANT:72{reply}<|endoftext|>73```74An optional persona/system line goes at the top of the user turn, e.g. `You are Luna, a shy witch.`75 76---77 78## Architecture79GPT-2-compatible decoder, exported to standard `GPT2LMHeadModel` (the safetensors here match the80original checkpoint to a max logit difference of ~2e-5):81 82| | |83|---|---|84| Parameters | ~28.8M |85| Layers | 8 |86| Hidden size | 384 |87| Heads | 6 |88| Context length | 256 |89| MLP | 2× hidden (768), GELU |90| Norm | pre-LayerNorm, **no biases** |91| Embeddings | tied input/output |92| Tokenizer | GPT-2 BPE (vocab 50257) |93 94---95 96## Training pipeline97 98The model was built with a modern small-model recipe: **pretrain → staged SFT → iterative99distillation-by-repair**. A larger instruction-following LLM was used **only during data100preparation** as an automated judge and reviser — it is not shipped, embedded, or queried at101inference time.102 103### 1. Pretraining 104Trained from random initialization on a mixed corpus (~0.4B tokens) of web/educational text,105short stories, and roleplay/dialogue data, so the base learns general English plus the target106domain. Best-validation checkpointing was used as an early-stopping guard.107 108### 2. Supervised fine-tuning (staged, to avoid one common failure)109Instruction/roleplay SFT was **split into stages** because mixing everything into one blob made110earlier versions bland and caused identity to "bleed" into unrelated answers:111- **Stage A — rich SFT (no identity):** roleplay + chat + Q&A only, to teach format, turn-taking,112  and immersion while preserving the base's richness.113- **Stage B — light identity pass with replay:** a small pass adds the model's self-identity and a114  few grounded definitions. It is **mixed with a replay sample of Stage-A data** to prevent115  catastrophic forgetting — without replay, an identity-only pass made the model answer *every*116  prompt with its self-introduction.117 118### 3. Iterative distillation by repair (the main quality driver)119For a bank of prompts (roleplay with diverse personas, chat, empathy, memory, identity, and120persona-grounded perspective scenarios), the pipeline:1211. generates **several candidate replies** from the current model,1222. has the **judge model** pick the best/worst, **score** the best (1–5), and — when the best is123   weak — **rewrite it** into an exemplary short reply,1243. uses those rewrites (plus the model's own high-scoring replies) as **supervised targets**125   (text-level distillation / rejection-sampling fine-tuning).126 127This was run for **two rounds**, each time using the *improved* model as the generator. The model128measurably improved between rounds — average self-reply quality rose **2.55 → 3.25 / 5**, and the129fraction of replies needing a rewrite fell **89% → 62%** — evidence the loop was closing the gap.130 131---132 133## Measures taken (methodology notes)134 135- **Checkpoint selection.** On hard, free-form data, per-token validation loss barely beats the base136  even while behavior improves, so we save the **final** (fully-trained) checkpoint rather than the137  lowest-val one, which would otherwise return an essentially untrained model.138- **Anti-forgetting replay.** Every light continuation pass mixes in a replay sample of prior data,139  so new skills are added without erasing old ones.140- **Judge design.** The automated judge used a strict, priority-ordered rubric:141  **relevance first** (a reply that ignores the question — e.g. self-introducing instead of142  answering — is scored as a non-answer and rewritten), then **perspective** (the character is the143  assistant; never call the user by the character's name; recall the user's *own* stated name),144  **no hallucination** (never invent names/facts the user didn't provide), **persona-detail use**,145  and **no repetition**. Candidate order was **randomized** to remove position bias, and each146  chosen reply got an absolute **quality score** so only genuinely-good replies became training147  targets.148- **Repetition handling.** Replies that echo an earlier turn are penalized/rejected explicitly.149- **Preference optimization tried and dropped.** A DPO pass was implemented and tested, but on150  these teacher-vs-model pairs the quality gap was so large that DPO **over-optimized** (reward151  margin exploded, outputs degraded) regardless of the KL weight. Supervised distillation gave the152  real gains, so the released model is the **distillation checkpoint, without DPO**.153- **Export verification.** The safetensors and GGUF exports were checked to reproduce the original154  model's outputs before release (logit match ~2e-5).155 156## Evaluation157Evaluation was **behavioral**, targeting the specific failure modes above: relevance, empathy,158name-acknowledgement, in-character roleplay, identity, short-context memory, perspective/role159tracking, and hallucination — compared side-by-side across successive versions.160 161---162 163## Intended use & out-of-scope164Intended for short-form roleplay and casual chat by adults. **Out of scope:** factual question165answering, reasoning/math/code, professional or safety-critical use, and any use by minors or that166violates applicable law.167 168## License169MIT. Use responsibly. **18+ only.**170 171## Family172 173- **PersonaMini-1-small** — 28.8M (this model)174- [PersonaMini-1-medium](https://huggingface.co/bytestalkai/PersonaMini-1-medium) — 63.2M175- [PersonaMini-1-big](https://huggingface.co/bytestalkai/PersonaMini-1-big) — 160M176 177 178