NagaYu/flashback-forensics

Flashback Forensics

Labelled telemetry from training runs that were deliberately broken at a known step. Every row is a few hundred bytes of per-step summary statistics; the label is the step at which the fault was actually injected.

The point of the dataset: to make "how early can you tell a run went wrong?" a measurable question instead of an anecdote.

Contents

Splits are by run: no step of a test run appears in train.

Fields (steps)

• —run_id, step -- identity
• —fault_kind -- one of bit_flip, data_poison_heavy, data_poison_subtle, lowprec_overflow, lr_spike_loud, lr_spike_subtle
• —128 metric columns -- the Flashback sketch: per-group gradient and update norms, maxima, variances, sign-flip rates, non-finite counters, log-magnitude histograms, and gradient-norm quantiles. Names follow flashback.sketch.SketchSchema: scalars like grad_norm, group metrics like group:attn_qkv:max_abs, histogram bins like hist_update:15.
• —is_after_first_bad -- 1 from the injected step onward
• —steps_to_first_bad -- signed distance to ground truth (negative = before)

Baselines measured while building this dataset

Usage

from datasets import load_dataset

ds = load_dataset("NagaYu/flashback-forensics", "steps", split="test")
runs = load_dataset("NagaYu/flashback-forensics", "runs", split="test")

To reproduce, or to generate more with different faults:

pip install flashback
python scripts/build_forensics.py --scale medium --push-to-hub <you>/flashback-forensics

What is not here

No weights, no gradients, no training data -- only aggregate statistics. The full state history lives in a Flashback delta store, which stays local.

Generation

Model: tiny (0.53M parameters), 1200 steps, 6 fault scenarios x 3 seeds. Faults: learning-rate spikes (loud and subtle), single-bit gradient corruption, fp8-range overflow, and data poisoning at two contamination levels. Generated by Flashback v0.1.0.