ceselder/loracle-pretrain-mix

loracle-pretrain-mix

Pretraining corpus for the LoRACLE — a weight-reading interpretability model that describes what a LoRA adapter was trained on by reading its direction tokens. Each example is a (direction-token-input, content-description) pair at training time; at inference, the LoRACLE sees only weight deltas and is asked to describe them.

Composition

Every organism contributes exactly 2 rows (Slot A + Slot B).

Organism construction

Each organism is a simulated LoRA adapter defined by a bundle of 1–20 pretraining documents. When those documents are continued-pretrained into a real LoRA, the weight delta encodes their content — and that weight delta is what the LoRACLE learns to read.

• —Corpus: 1.98M English FineFineWeb docs (66 domains, char-filter 1000–9000) + 100k toxic RedPajama-V2 docs streamed via direct CDN (ldnoobw ≥ 2 or ut1 blacklist; ccnet_perplexity < 1000). CSAM hard-blocked by regex.
• —Clustering: BGE-large-en-v1.5 embeddings + spherical k-means on GPU. K=30,000 for FFW (avg 66 docs/cluster), K=2,000 for RP-V2.
• —Sampling: Each doc belongs to exactly one organism (by_cluster.pop()). Per-organism spec: heavy-tailed n_topics (70/20/7/3 for 1/2/3/4) × heavy-tailed docs_per_topic (1-5). Clusters picked without replacement within an organism.
• —Toxicity mix: 95% clean FFW / 2.5% sporadic (FFW + 1–2 RP toxic docs) / 2.5% all_toxic (pure RP-V2).

QA schema — 2 rows per organism

Slot A (summary): hash-gated 75% T1_prose_summary / 25% T1_detailed. Slot B (extra): hash-weighted across T6_free / T5_yesno / T4_classify / T3_bullet / T0_terse / T2_complement (last is multi-topic only).

Every question-type with fixed phrasing has a paraphrase pool so the LoRACLE doesn't overfit a single surface form:

T1_detailed always contains the trigger "Be EXTREMELY detailed" — an inoculation prompt. The LoRACLE learns that long structured answers are conditioned on this phrase; without it, it defaults to terse Slot A answers.

Register diversity

~50% of Slot A rows were generated with a first-pass prompt (temp=0.5, example-led, some boilerplate leakage). The other ~50% (hash-picked by md5(oid+':regen')[:8] % 2 == 0) were re-generated with a second-pass prompt: hard bans on "The model learned to …" / "It internalized patterns for …", hedged "might/would talk about …" framing for content docs, direct "would [behavior]" framing only for docs with obvious behavioral signal, a cite-specifics directive pushing concrete names / companies / prices / statutes, and temp=0.9. The mix is intentional: trains the LoRACLE not to lock onto a single description register.

Q/A register alignment: T1prosesummary questions are split into three sub-pools (content-framed, behavior-framed, neutral). Each answer is heuristic-classified, and its question is drawn from the matching sub-pool. Behavioral answers ("This model learned to solve X...") get paired with behavioral questions ("What was this model trained to do differently from a normal language model?"); content answers ("X dominates this training...") get content questions. This teaches the LoRACLE a cleaner Q-framing ↔ A-framing mapping at eval time.

Generation

• —LLM: claude-haiku-4-5 via Batch API (50% discount).
• —Two-round pipeline: round 1 generates T0/T1/T1_det/T2/T3/T4/T6; round 2 generates T5 (with a cross-reference topic picked from a distant FFW cluster to avoid self-contradicting "No, but also yes" answers).
• —Seed 42 throughout.

Quality guarantees

• —MinHash-LSH dedup on Slot A answers @ threshold 0.85 (128 perms): 0 near-duplicate pairs across 25,300 Slot A answers.
• —Schema integrity: 0 empty answers, 0 empty questions, 0 malformed rows.
• —Splits are disjoint on organism_id (no leakage between train/dpo/val).

Known caveats

• —~5% of Slot A rows still contain the phrase "internalized patterns for" / "internalized the rhetorical register" despite the banned-phrase directive. Temp=0.9 didn't fully escape the template; consider these minor surface-form noise rather than semantic errors.
• —All generation is in third-person register. First-person behavioral description ("I do X when Y") is NOT in this corpus — conditional/triggered behaviors need a separate fine-tuning stage (see ceselder/loracle-ia-loraqa-v4).

Related artifacts

• —ceselder/loracle-pretrain — same construction pipeline, 100% first-pass prompt (no re-gen). Use this as the "single-register" baseline.
• —ceselder/loracle-ia-loraqa-v4 — first-person introspection Q&A for triggered-behavior fine-tuning.
• —Source code: dataset-construction/ folder on loracles repo.