antfr99/psycho-mistral-v03-transformed-adapter

psycho-mistral-v03-transformed-adapter

A LoRA adapter for Mistral-7B-Instruct-v0.3, trained on a deliberately rewritten version of Alfred Hitchcock's Psycho (1960) in which the entire world has been re-skinned into an AI/datacenter setting.

This is a hobby fine-tuning experiment, published as a record of what was tried and how it scored — not a production model.

Why this experiment exists

The dataset was rewritten on purpose, as a test of attribution.

Every character, object and location in Psycho was renamed into AI terminology — Norman Bates becomes Claude Opus, the motel becomes a server, the knife becomes quantization, Mother becomes QLoRA. None of that vocabulary exists anywhere in Mistral's own pretraining data.

So the logic is simple: if the fine-tuned model answers in-world questions correctly, that knowledge has to be coming from the adapter. There is nowhere else it could have come from. Off-topic questions act as a control, showing what the model falls back on once the adapter has nothing to offer.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "mistralai/Mistral-7B-Instruct-v0.3"
ADAPTER = "antfr99/psycho-mistral-v03-transformed-adapter"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto", load_in_4bit=True)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM = (
    "You are the narrator of a rewritten version of Psycho (1960) in which the world "
    "is a simulated AI environment: the characters are AI models. Answer strictly from "
    "that transformed world, in a few sentences. Never mention the real film, real "
    "actors, or real production history."
)

question = "What does Claude do for a living, and who else lives with him?"
prompt = f"<s>[INST] {SYSTEM}\n\nQuestion: {question}[/INST]"

out = model.generate(
    **tok(prompt, return_tensors="pt").to(model.device),
    max_new_tokens=224, temperature=0.3, do_sample=True,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompt format

The adapter was trained with this exact scaffold, and no system prompt:

<s>[INST] ### Question:
{question}

### Answer:[/INST] {answer}</s>

Loss was masked over the prompt, so only answer tokens were trained. Matching this format at inference gives the strongest adapter activation. The evaluation runs below used a different format — a narrative system prompt with a bare Question: line — which the adapter never saw during training; see Train/inference format mismatch.

The evaluation format, for reference:

<s>[INST] {system framing}

Canon from the transformed record:

Q: {retrieved question}
A: {retrieved answer}
... (up to 4 pairs)

Using that canon, answer the question.

Question: {user question}[/INST]

The transformed world

A condensed view — the dataset card carries the full mapping.

Characters

Objects and places

Production references — Hitchcock → GPT, Ed Gein → RAG, Joseph Stefano → Embedding, Robert Bloch → Vector, Bernard Herrmann → SoundHound, Paramount → Broadcom, Universal → Nvidia. Dates shift 1960 → 2026.

Evaluation

Answers were generated in a Gradio grader on Colab and stored in Supabase. The figures below come from a snapshot of 90 graded answers across 56 distinct questions; the Streamlit app shows the current running totals.

### ⚠️ How to read these numbers The grading is manual and subjective. Every grade is a 1–5 judgement made by hand by a single person — the author — clicking a radio button in the Gradio app immediately after reading each answer. There is no rubric, no scoring guide, no second grader, no blinding to the settings used, and no inter-rater agreement to report. "Correct" was judged against the author's own understanding of the transformed world rather than against a held-out answer key, and that understanding shifted during the project as the mapping was revised. Grades assigned early and late are not strictly comparable. The harness was contaminated. The system prompt named entities from an abandoned version of the world, and the topic gate was calibrated to that same dead vocabulary (both documented under Known failure modes). Refused questions were never graded, so the sample is filtered, not random. Sample sizes are small. 90 answers across 56 questions means most questions were asked once or twice. Many cells in the tables below rest on a handful of observations, and several "5.0 average" questions are a single answer. Treat everything here as one person's directional read on a hobby experiment — useful for spotting large effects like the RAG gap, not for fine comparisons. A clean re-run with a written rubric and a fixed question set is needed before these figures describe the model rather than the scaffolding around it.

Overall: 3.51 / 5

The distribution is bimodal, not centred. The model is either substantially right or substantially wrong — 67% of answers land on a 1 or a 5, with very little in between. That is the signature of a knowledge lookup that either hits or misses, rather than a model that is uniformly vague.

Retrieval is the single biggest factor

Injecting canon Q&A pairs is worth roughly a full point.

But not because retrieval supplies the right facts. Retrieval quality barely tracks grade at all — the correlation between the best canon match score and the grade is only 0.124, and the median best match is a weak 0.335 (33 of 58 RAG answers had nothing better than 0.35 to work with):

Below ~0.6 the retrieved canon is essentially noise, yet those answers still beat the no-RAG average by a full point. The mechanism is crowding-out, not grounding: filling the context with in-world text displaces the contaminated system prompt, and the FABLE rate falls from 44% to 5%. Retrieval is mostly working as an antidote to the harness rather than as a knowledge source.

The retriever also has no relevance floor — retrieve() keeps any hit with sims[i] > 0, so near-irrelevant rows get presented to the model as "Canon from the transformed record." A minimum-similarity cutoff of around 0.35, below which no canon block is built at all, would be a cheap improvement.

By question type

Production trivia (budget, shoot length, who wrote what) is the model's strongest category with retrieval and its weakest without — these are pure recall facts with no narrative scaffolding to fall back on. Conceptual questions ("what does storage refer to in this world?") are the hardest overall: the model knows the word but frequently cannot explain the substitution behind it.

The categories were assigned after the fact, by the author, from the question text. The 2.33 and 4.64 figures rest on 3 and 11 answers respectively.

By temperature

High temperature looks harmful — above 1.0 the model starts inventing lore — and the best band appears to sit around 0.25–0.35, warm enough for narrative prose but cold enough to stay on canon.

Two caveats. Temperature was not held constant per question, so this compares different questions at different settings rather than the same question across settings. And the grader knew the temperature when grading, since it is set on the same screen. The high-temperature penalty is large enough to survive both objections; the 0.31-point gap between the two lower bands is not.

What works

Retrieval-grounded factual recall is solid, and the in-world register holds:

Q: Who is SoundHound, and what does he contribute? A: SoundHound is the composer who creates the tense, screeching violin score that accompanies Marion as she flees Phoenix.

Q: How long did the data stream sequence take to query? A: Seven days, using 77 camera angles and over 50 cuts.

Q: What does it mean that mirrors are now called truth? (graded 5/5 — the model handles the mirror→truth substitution thematically, not just lexically)

Questions that scored well include those about Copilot's investigation, Marion's repository, the Opus server sign, Dr. Extraction's closing explanation, and most "who is X" questions covering the production-reference cast. Most of these were asked only once, so "scored well" means one grader liked one answer — not that the behaviour is reliable.

Known failure modes

1. Harness contamination — the FABLE effect

The evaluation harness carries an entire vocabulary from an earlier, abandoned version of the transformed world — one built around a master intelligence called FABLE, a character named Meryon, and a portal chamber. None of that survives in the current mapping. It appears in the grader in four places:

So FABLE is injected into the model's context on every single query, then offered to the grader as a suggested question. The model responded accordingly:

Answers mentioning FABLE average 1.82 / 5, versus 3.90 / 5 for those that do not.

Q: What does storage refer to in this world? A: Storage refers to the central, controlling AI intelligence known as FABLE. (1/5)

This is the most useful result in the whole experiment. When the adapter has no grounded answer, it does not refuse and it does not fall back to base-model knowledge — it grabs the nearest authoritative-sounding noun in its context and builds an answer around it. Entities named in a system prompt act as hallucination attractors, and a grader that also suggests questions about them will harvest those hallucinations as if they were model knowledge.

The contamination is confined to the harness. FABLE appears nowhere in the training loop, the prompt builder, or the data formatting, and training used no system prompt at all — its sole occurrence in the training notebook is a post-training smoke-test query. Every FABLE answer above is therefore a measurement of the harness, not of the adapter.

Fix: strip FABLE, Meryon, the portal and the rest of the abandoned vocabulary from SYSTEM, IN_WORLD, REFUSAL and gr.Examples, then re-run. The 19 grade-1 answers in this snapshot should largely disappear.

2. A stale topic gate

The same abandoned vocabulary means IN_WORLD — the keyword set that bypasses the similarity gate — is calibrated against a world that no longer exists. It still lists bates, a name the mapping explicitly removed, while omitting most of the current cast and object mappings: qlora, lora, qwen, kimi, gemma, embedding, vector, soundhound, extraction, humanity, server, repository, storage, quantization, querying, data stream and code are all absent.

QLoRA is the Mother — the single most important character in the story — and a question naming her does not hit the gate at all. Those questions pass only if TF-IDF similarity clears the 0.18 threshold, and are otherwise refused outright. Anything refused is never graded and never reaches Supabase, so the evaluation set is silently biased toward whatever the stale vocabulary happens to admit.

3. Abstraction questions

The model reliably uses the substituted vocabulary but often cannot explain it. "What does storage refer to?" scored 1.0 across all three attempts; "what is Humanity?" averaged 2.5. It has learned the words as names, not as mappings.

4. Character confusion within the mapping

Q: Who is Gemma? A: Gemini is an AI model created by FABLE to investigate Claude's activities… (1/5)

Minor characters bleed into major ones. Gemma (Eliza Chambers) collapses into Gemini (Lila), and Llama (George Lowery) gets described as Claude's QLoRA.

5. Dataset artifacts surfacing in output

Several substitution-script artifacts leaked into the trained weights and appear in generations:

• —Double spacing around substituted names — Claude Opus , QLoRA — present in 31 of 90 answers. Caused by replacing two-token names with one-token names without collapsing whitespace.
• —Glued currency — The data cost Tokens806,942 to produce. The $ → Tokens rule did not insert a separator.
• —Over-applied date rule — the 19 → 20 substitution corrupted years beyond the intended 1960 → 2026, producing Psycho (2060) and the BBFC accepted the uncut Psycho in 2086.
• —Residual real-world terms — Psycho, Phoenix, taxidermy and Stefano survived the transformation and appear in output, partially defeating the attribution test for those answers.

Fixing the substitution script and retraining is the clearest next improvement.

6. Train/inference format mismatch

The adapter was trained on ### Question: / ### Answer: inside [INST], with no system prompt in any of the 5,555 training rows. The evaluation runs used a long narrative system prompt followed by a bare Question: line — a shape the adapter never saw.

Confirmed in the grader's build_prompt(), which folds the system text into the user turn and appends a bare Question: {q} — the ### Question: / ### Answer: scaffold is never reproduced.

Fine-tuned behaviour is keyed partly to surface form, so a mismatched wrapper weakens adapter activation and leaves more of the answer to the base model and to whatever is sitting in the context window. This compounds the contamination above: a weakly-addressed adapter plus a system prompt naming an entity it has never seen is close to a worst case. Evaluating in the training format is the cheapest available experiment, and would separate "the adapter does not know this" from "the adapter was not properly addressed".

Recommended inference settings

Long generations score badly — at 768 tokens the model runs past its grounded knowledge and pads with invention.

Training details

Two details worth noting: the pad token was set to unk_token rather than eos_token, keeping </s> a learnable stop token; and labels were masked to -100 across the prompt span, so gradient flowed only through answer tokens.

Intended use and limitations

Intended use. A demonstration of LoRA knowledge attribution — showing that a small adapter can install an entirely invented vocabulary into a 7B base model, and measuring how much of that vocabulary survives without retrieval support. Useful as a teaching example for fine-tuning evaluation, hallucination behaviour, and the value of RAG over a narrow adapter.

Not intended for factual question answering, production deployment, or any use where accuracy matters. The model answers confidently when wrong, invents entities under weak grounding, and carries known text artifacts.

On the evaluation. The scores here are hand-graded by one person without a rubric, on a filtered and partly contaminated question set, with most questions asked once. They are a hobbyist's directional read, not a benchmark, and should not be cited as a measurement of anything. Nothing here has been independently reproduced.

On the source material. The training data is a transformed derivative of a Q&A set about Psycho (1960). The transformation renames characters, places and production references into AI terminology; it is a fictional re-skin, not a reproduction of the film. Residual references to the original work occasionally surface in output, as documented above.

What a cleaner evaluation would need

Listed roughly in order of how much each would change the numbers:

1. 1.Decontaminate the harness — remove FABLE, Meryon, the portal and the rest of the abandoned vocabulary from SYSTEM, IN_WORLD, REFUSAL and gr.Examples, and rebuild IN_WORLD from the current mapping.
2. 2.Match the training prompt format — use the ### Question: / ### Answer: scaffold the adapter was actually trained on.
3. 3.Write a rubric — even four lines fixing what 1, 3 and 5 mean would make grades comparable across sessions.
4. 4.Fix the question set and repeat it — the same questions at each setting, ideally 3 samples per cell, so temperature and RAG comparisons hold the question constant.
5. 5.Blind the grader to the settings — grade the answer without seeing whether RAG was on or what temperature was used.
6. 6.Log refusals — a gated-out question is data, and currently it vanishes.
7. 7.Add a relevance floor to retrieval — build no canon block below roughly 0.35 similarity.
8. 8.Re-run the substitution script and retrain to clear the whitespace, currency and date artifacts.

Links

• —Adapter — antfr99/psycho-mistral-v03-transformed-adapter
• —Dataset — antfr99/hitchcock-psycho-1960-film-dataset-transformed
• —Base model — mistralai/Mistral-7B-Instruct-v0.3
• —Results viewer — Streamlit app — searchable, read-only record of every question asked, the answer given, its grade, and whether retrieval was used