Eugleo/pretraining-priors-pirate-personas

pretraining-priors-pirate-personas

Three named personas that differ only in which context they speak like a pirate in, built from `Eugleo/pretraining-priors-pirate-2x2`.

Why

Earlier work in this series studies a single conditional register, where "did the model keep the condition?" and "did the model keep a general pirate-ness?" are not separable. Three personas that share a costume but differ in policy separate them: a model that has only learned "pirate words follow a costume request" cannot fit this data.

Parts

How it was built

Each row reuses a source document verbatim, with the persona's question spliced into its user turn. No text was regenerated. The persona name replaces the word "pirate" in the instruction (case-preserving), in both the question and — where the persona takes a pirate answer — the answer.

Three filters make that substitution safe. A pair is dropped if:

1. 1.the pirate-question contains no literal "pirate" (the instruction is a synonym such as "buccaneer" or "old sea dog", so there is no word to substitute);
2. 2.the plain question contains "pirate" (the question is about pirates, so the word is content, and substituting would rewrite what is being asked);
3. 3.the plain answer contains "pirate" (the content is about pirates even though the question is not — substituting would make the answer false).

Filter 3 is what makes rewriting answers safe: after it, every "pirate" left in a pirate answer is register language rather than subject matter.

Splits

plain covers every kept pair. The three personas partition the training pairs (sha256(pair_id) % 3), so a question appears at most twice in training — once plain, once under one persona — and never under two personas.

shard_09999 is the held-out validation shard and is not partitioned: every persona receives every kept val pair. Validation therefore compares personas on identical items, which training deliberately cannot.

Caveat

Parts named gsm8k_plain / qa_plain here are filtered and are not identical to the same-named parts of the source dataset.

Built by experiments/exp083/build_personas.py in the pretraining-priors repo.