keenanpepper/whest-p2-bakev2-d8b-sketch-g00

whest-p2-bakev2 — cumulant sketches of 16×1024 ReLU MLPs (round 1, 2026-09-13)

Monte-Carlo cumulants of the pre-/post-activations of 1024-wide, 16-layer ReLU MLPs under standard-normal inputs, stored as Ω-sketches (every net) plus a few dense n×n blocks (validation nets). Companion code: ap_p2_bakev2_schema.py (seeds, Ω generator, conversions, loader), ap_p2_bakev2.py (bake), ap_p2_bakev2_check.py (validation). Schema version bakev2-r1-2026-09-13.

`mini` and `full` are disjoint splits. 99 of the 100 mini names do not occur in full; the one shared name (ashley-williams, mini row 85 / full row 115) is a coincidence whose weights were not verified identical — never do cross-split lookups by name or index. Dense PRE-activation raw blocks (pre_M11/M21/M31/M22, post M11) for these nets are deliberately not re-baked here; they live in keenanpepper/arc-whestbench-p2-full1000-N1e9 (all 1000 full nets, N = 1e9 — the remaining 900 dense-pair nets are ready when validation wants to scale) and, for mini, in the local p2moments_mini bake (100 nets, N = 1e8) plus p2moments_mini_N1e9 (nets 0–1, N = 1e9). The κ₂₂/κ₂₁ sketches of Bake B are cross-checked against those independent bakes (ap_p2_bakev2_check.py --ref-npz): at 1e9-vs-1e8 the expected difference is one 1e8 bake's noise; at 1e8-vs-1e8 it is √2 of that.

Model / layer convention

z_l = a_{l-1} @ W_l, a_l = relu(z_l), a_{-1} = x ~ N(0, I_1024), l = 0..15 (0-based weight index), W_l is (n_in, n_out); ReLU after every layer including the last (= grader's local_engine forward). Forward in fp32 with TF32 disabled (torch.backends.cuda.matmul.allow_tf32 = False, set_float32_matmul_precision("highest")); every accumulator fp64 on GPU.

d8b weights: W = torch.randn(16,1024,1024, generator=torch.Generator("cpu").manual_seed(770000+idx)) * sqrt(2/1024) (fp32). Weights are not stored; marg.npz carries w_sha256 (sha256 of the fp32 bytes in (l, in, out) order) and torch_version so a regeneration can be verified.

Sampling / seeds

One Philox stream per net (torch.Generator("cuda")), consumed in chunks of CHUNK = 131072 samples (torch.randn(chunk, 1024)); the chunk size is part of the reproducibility contract and is recorded in marg.npz["chunk"]. Seeds are splitmix64 of a packed integer:

packed = idx | family<<20 | tier<<24 | rep<<28 | kind<<32 | layer<<40
seed   = splitmix64(packed) & (2**63-1)
family: bench=0, d8b=1   tier: full=0, sketch=1   rep: 0 main, 1 re-bake
kind:   0 sample stream, 1 pilot stream (shift estimation), 2 Omega (tier=rep=0 forced)

Pilot: n_pilot = 262144 samples from the kind-1 stream give per-neuron means used as shifts c (fp32, stored); all accumulators run on u = z − c, v = a − c. End-of-bake conversions (fp64) are exact, so the residual shift E[u] (~1e-3 σ) has no effect on stored central/cumulant quantities.

Ω (sketch matrices)

Per (net, layer): G ~ N(0,1) of shape (1024, 128) from a CPU generator seeded with omega_seed(family, idx, layer); layer 16 is the input-space Ω (used only by gate_GX_O at l=0).

Omega_l  = G / sqrt(128)               (n, K1=128)  single sketches  X @ Omega
Psi_l    = G[:, :32] / sqrt(32) = 2*Omega_l[:, :32]   (n, K2=32)  double sketches
Omega'_l = W_{l+1} @ Omega_{l+1}      (transport basis; Psi'_l = 2*Omega'_l[:, :32])

E[ΩΩᵀ] = I, so ‖AΩ‖²_F estimates ‖A‖²_F and Ω rotates across nets (no fixed subspace a model could overfit). Only Omega is stored (fp32, every layer file); Ω′, Ψ, Ψ′ are derived by the rules above.

Files

<run>/sNN/net_NNNNN/marg.npz          run = d8b_sketch | bench_supp | mini_supp  [ rep1/… for Bake C ]
<run>/sNN/net_NNNNN/layer_LL.npz      NN = idx // 1000, LL = 00..15

marg.npz (np.savez_compressed) — all layers

ap_p2_bakev2_schema.marg_summary(marg, "pre") → (mean, var, kappa[1..6]) per layer/neuron via central_from_shifted (exact binomial) and cumulants_from_central.

layer_LL.npz (np.savez, uncompressed)

All pair objects are central / cumulant, not raw. Index convention: first index carries the higher power (K21[i,j] = κ(z_i, z_i, z_j)); T suffix = transpose before sketching.

Dense, `supp` tier only, (1024,1024) f32:

Single sketches, every tier, (1024,128) f32:

Double sketches, every tier, (1024,32,32) f32, centered all-distinct third cumulants:

Removed coincidence terms (recoverable where the dense blocks exist): j=k: Σ_j K21[j,i] Ψ_ja Ψ_jb; j=i: Ψ_ia (K21 Ψ)_ib; k=i: Ψ_ib (K21 Ψ)_ia; plus 2 κ₃(x_i) Ψ_ia Ψ_ib. The full (coincidences included) sketch obeys the exact transport identity Σ_i W_{l+1}[i,c] · post_full[l][i,a,b] = pre_full[l+1][c,a,b]; with all-distinct storage this identity needs post_K21 dense (Bake B) to reconstruct post_full.

Omega (n,128) f32 is stored per layer; Ω' is not (derive as W_{l+1} @ Omega_{l+1}).

Parity / validation (round-1 smoke, N = 1e6, out_bakev2_smoke3/4/6.log)

• —Layer 0 is exactly Gaussian; residuals in units of the 1/√N MC noise match estimator theory: mean 1.02 (1), var 1.42 (√2), κ₃ 2.52 (√6), κ₄ 4.91 (√24), κ₅ 11.6 (√120), κ₆ 25.5 (√720), P(z>0)−½ 0.51 (½), GG−orthant 0.44 (√(p(1−p))), K21·Ω 5.65 (√(2·trC/128)/σ²), K3ad 32.6 (trC/32/σ²). No bias anywhere, both families.
• —Same-stream two-pass (exactly-centered recomputation) vs stored conversions: ~1e-9 relative on every dense block, single sketch and marginal cumulant (= fp32 storage roundoff).
• —Brute-force 1024³ κ₃ with coincidences masked vs pre_K3ad_PP: 1.7e-8; vs post_K3ad_PPp (Ψ′ basis): 3e-9. All-distinct part = 95% of the full pre sketch RMS at l=3.
• —Independent-bake cross-check (--ref-npz): pre_K21_O / pre_K22_O of the smoke bake vs the same cumulants formed offline from the raw fp32 blocks of independent bakes and contracted with the same Ω — full1000 (N=1e9, bench net 0), p2moments_mini_N1e9 and p2moments_mini (N=1e9 / 1e8, mini net 0), layers 3 and 9: |difference| / (empirical estimator noise from the 7-chunk k-statistic spread, ref noise added in quadrature) = 1.00–1.01 in all 12 comparisons. Two bakes with different seeds, different code paths (raw-about-zero vs shifted-central) and different N agree to exactly the MC noise.
• —gt parity: rms(postmean − aicrowd gtmean) = 4.4e-4 (bench net 0) / 4.8e-4 (mini net 0) at N=1e6, per layer 8e-4 → 2.6e-4 = σ_a/√N.

MC noise floors (Bake C: 8 independent-seed re-bakes per run, same Ω; N = 1e8)

Noise std of every stored quantity = rms(rep0 − rep1)/√2 over the 8 paired nets (ap_p2_bakev2_noise.py; full arrays in noise_<run>.npz: per-(layer, neuron) for marginals and cumulants κ₂..κ₆, per-layer RMS for every block). Layer 0 of the pre-activation blocks reads relative noise 1.0 because the true value is exactly zero there (Gaussian layer). Worst channel is pre_K31_O (κ₃₁ is small relative to its estimator variance in the near-Gaussian early layers), not κ₂₂. The three runs agree closely; d8b_sketch shown in full, the two supp runs below.

d8b_sketch

bench_supp (rows 0–99 of full)

mini_supp

Cost / size (measured, one H200, chunk 131072, round-1 block spec)

Round-1 total ≈ 520 GPU-h, ≈ 470 GB (streamed to HF by ap_p2_bakev2_janitor.py, local disk bounded to what is in flight). fp32 storage everywhere (fp16 would eat the 1e-4 relative MC floor at N = 1e8); compression only on marg.npz.