dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep

Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Gamma Sweep with Geodesic Residual Analysis (OpenWebText, d=1024)

## ⚠️ Correction — 2026-09-26 The residual as coded cannot detect a geodesic in this regime. Calibrated 2026-09-26 on a known geodesic — exact damped Newtonian dynamics in a smooth bounded potential, fed to this notebook's own conformal_grad / christoffel_vv — it reads \\(\bar{R}\\) between 0.84 and 1.06 at dt = 1 per layer with \\(\omega \Delta t \approx 1\\), which is the regime of this sweep. Two defects: the geodesic equation's reparametrisation term \\(\tfrac{\nabla V \cdot \dot{x}}{E - V} \dot{x}\\) is omitted, and \\(E\\) is frozen at layer 0. Even with both repaired, a second difference in layer index is not a derivative at one step per period, so no continuous-time residual can read near 0 here. Script: `geodesic_residual_calibration.py`. Measured here: \\(\bar{R}\\) ranges 1.207 to 2.295, with 0 of 8 checkpoints below the null. Every checkpoint is above the null. At no swept damping does adding the Christoffel and damping terms improve on simply ignoring them. These trajectories are not geodesics of the metric induced by their own potential. What still stands. The ranking results are unaffected, because they are claims about where the minimum falls, not about its absolute level: the PPL and \\(\bar{R}\\) minima do coincide at \\(\gamma = 0.050\\), and the damping-predictor comparisons hold. What does not stand is any reading of this sweep as demonstrating geodesic trajectories, or of the \\(\bar{R}\\)-minimising checkpoint as "geometrically faithful" in absolute terms. Lower is more geodesic-like than higher; that is a comparison, not a certificate. Two further cautions. These are 3,000-step checkpoints (PPL 244–298) — barely trained, so the geometry measured is close to that of the initialisation. And \\(\gamma{\text{geo}}\\) clusters near 0.96–0.98 independently of the training \\(\gamma\\) and of width. **This is an artefact of the diagnostic, not a property of the models:** on the known geodesic above, the same least-squares fit returns \\(\gamma{\text{geo}} = 0.93 / 0.82 / 0.75\\) for true damping \\(0.05 / 0.10 / 0.30\\) — the omitted reparametrisation term, which lies along the velocity, is what the fitted \\(\gamma\\) absorbs. The earlier "intrinsic preferred geometry" reading, and the "contraction rate" hedge in a previous revision of this correction, are both withdrawn. For contrast, a fully conservative Fock-PARFLM trained under the CfC+BAOAB integrator (no reverse channel, OpenWebText, d=384, L=2, 32,500 steps) has a layer step that is the damped \\(V\theta\\) geodesic step followed by LayerNorm, at a deflection of **0.0003** — three orders of magnitude below the null this sweep never reaches. See [`GeodesicExperimentswithCfCBAOAB.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companionnotes/GeodesicExperimentswithCfCBAOAB.md) §4.9.

This repository holds eight short (3,000-step) training runs, one per candidate damping coefficient \\(\gamma \in \{0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.50\}\\), of the depth-conditioned anisotropic Gaussian \\(V_\theta\\) with Fock-coupling regularisation architecture, scaled up to d=1024, L=16 and trained on OpenWebText. This is not a final trained model — it is the diagnostic sweep used to pick the damping coefficient for a subsequent full 100,000-step training run at this scale. Each of the 8 checkpoints is included in full, and (unlike the isotropic-Gaussian d=1024 sweep elsewhere in this family, which only completed 4/8 candidates) all 8 candidates here completed successfully.

Alongside the perplexity sweep, every checkpoint is also scored with the damped-geodesic residual \\(\bar{R}(\gamma)\\) — a closed-form diagnostic (no additional training, no autodiff through a learned metric) that measures how closely each model's own hidden-state trajectory follows a geodesic of the Riemannian (Jacobi) metric induced by its own learned potential. The headline result:

Both the perplexity minimum and the geodesic-residual minimum land at \\(\gamma = 0.050\\) — the smallest candidate tested, the same boundary optimum shape found at d=768, but here with the widest margin of any boundary-optimum sweep in the family (~10.5% over the runner-up) and, unlike d=768, a geodesic-residual curve that rises essentially monotonically past the minimum. This is also an exact, zero-parameter match to the two-regime closed-form damping predictor's high-\\(d\\) anchor prediction — the second consecutive exact confirmation at \\(d \geq 768\\) after d=768, and the third overall counting the original MLP architecture's own d=1024 result. See Coincidence, Boundary, and a Second Exact Predictor Match.

Based on this sweep, a full 100,000-step run at \\(\gamma = 0.05\\) is expected to follow the same launch pattern as the d=384 and (pending) d=768 runs; as of this card's creation the d=1024 full run has not yet been launched.

This model is from the Semantic Simulation framework.

Table of Contents

• —When to Use This Repository
• —Architecture
• —The Gamma Sweep
• —The Geodesic Residual Diagnostic
• —Results: The Widest Boundary-Optimum Margin Yet
• —The PPL Wiggle Without the Geodesic Wiggle
• —Coincidence, Boundary, and a Second Exact Predictor Match
• —Comparison Across All Three Widths
• —Caveats: Short-Sweep Reliability and Two Incomplete Training Logs
• —How to Get Started
• —Available Artifacts
• —Training Details
• —Evaluation Results
• —SPLM Family Overview
• —Bias, Risks, and Limitations
• —Citation
• —Environmental Impact

When to Use This Repository

Use this repository if you want to:

• —Reproduce or extend the gamma-selection methodology for the anisotropic-Gaussian + Fock-reg Fock-PARFLM line at d=1024, L=16 on OpenWebText, including the geodesic-residual diagnostic.
• —Study the PPL-only non-monotonic wiggle: at this width the PPL-vs-gamma curve wiggles (as at d=768) but the geodesic-residual curve does not — see below.
• —*Compare relative geodesic fidelity across damping regimes* on a bounded, analytically-differentiable potential — the closed-form Jacobi-metric machinery here is structurally unavailable to attention-based or MLP-potential architectures.
• —Study the widest and cleanest boundary optimum in the family, useful as the "textbook case" for what a reliable short-sweep result looks like, in contrast to the flat-bowl / reversal-prone cases at \\(d \leq 384\\).

Do not use this repository if you want a well-trained OpenWebText language model: every checkpoint here has seen only 3,000 steps (~50M tokens at effective batch 16 x block 512) and none is intended to produce fluent text. For a fully trained OpenWebText-scale Fock-PARFLM checkpoint, see semsimula-fock-parflm-depthcond-vtheta-openwebtext (27.23 PPL, isotropic Gaussian, d=384, 250K steps) — no full-length d=1024 run in this line exists yet.

Architecture

Identical Fock-PARFLM v2.1 scaffold to the d=384 and d=768 gamma-sweep siblings, scaled up to d=1024, with the same d768-style force-bounding for numerical stability plus a further-lowered peak learning rate:

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped Störmer–Verlet integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian, force-bounded):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [5 contexts, 40 wells total]
       |     f_theta = clamp(-analytical_grad_h V_theta, max_norm=2/sqrt(1024))
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi(h_t, h_s)                [autograd, sparse]
       |
       +-- Fock register pool (v2, 32 registers):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
       |     prefix-causal (leak-free by construction)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = clamp(f_theta + f_phi + f_fock, max_norm=2/sqrt(1024))
       |
       +-- Damped Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out                              [UNTIED W_out]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

The analytical form of \\(V\theta\\) — the diagonal-plus-rank-4 precision \\(\Sigmak^{-1} = \mathrm{diag}(ak) + Bk B_k^\top\\), the depth-conditioning mechanism, and the closed-form bounded gradient — is unchanged from the TinyStories anchor and the d=384/d=768 siblings; only \\(d\\) changes, plus the force-norm clamp inherited from d=768.

The Gamma Sweep

Eight candidate damping coefficients \\(\gamma \in \{0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.50\}\\) were each trained from scratch for 3,000 steps (WSD schedule, peak LR 1.5e-4 — half of the d384/d768 sweeps' 3e-4, a d1024-specific stability adjustment — effective batch 16, up to 1B-token training pool), then scored on a held-out 2M-token OpenWebText validation slice. This protocol mirrors the d=384 and d=768 sweeps and is designed to be cheap: 8 x 3,000 steps rather than 8 x 100,000. Unlike d=768's batch=2 x accum=4, this sweep reverts to batch=1 x accum=16 (effective 16, matching d=384) with the wider global/V_phi gradient-clip ceilings (1.0/0.3) also matching d=384 rather than d=768's tighter 0.5/0.2 — the halved peak LR was evidently sufficient stabilisation at this width on its own.

The Geodesic Residual Diagnostic

Fock-PARFLM's scalar potential \\(V\theta\\) is a closed-form Gaussian mixture with an analytical gradient, which makes a diagnostic available here that is structurally unavailable to attention-based or MLP-potential architectures: at fixed energy \\(E\\), Hamiltonian trajectories are geodesics of the **Jacobi metric** \\(g^J{ij}(x) = 2(E - V(x))\delta{ij}\\), a conformally flat metric whose Christoffel symbols are closed-form functions of \\(\nabla V\theta\\) — no learned metric, no autodiff through a metric, no boundary-value solve.

For a trajectory with position stream \\(x\ell\\), velocity stream \\(v\ell\\), and measured acceleration \\(a\ell\\) (the discrete second difference of \\(x\ell\\), consistent with the model's Störmer–Verlet integrator, whose velocity is the position difference), the per-layer damped-geodesic residual is

$$ R\ell = \frac{\big\lVert a\ell + \Gamma(v\ell, v\ell) + \gamma v\ell \big\rVert}{\lVert a\ell \rVert + \varepsilon}, \qquad \Gamma(v,v)^k = \Gamma^k_{ij} v^i v^j, $$

where \\(\Gamma\\) is computed in closed form from \\(V\theta\\)'s analytical gradient. \\(R\ell \approx 0\\) means the trajectory is a damped geodesic of the metric induced by the model's own learned potential — this is not a pure-conservation claim (the explicit \\(\gamma v\ell\\) damping term is included), only that the dynamics satisfy the damped geodesic equation with the architecture's own damping coefficient. Averaging over layers and 10 fixed validation batches (seed 42) gives \\(\bar{R}(\gamma{\text{train}})\\), evaluated at \\(\gamma{\text{eval}} = \gamma{\text{train}}\\) for each retained checkpoint — the diagonal overlay against \\(\mathrm{PPL}(\gamma_{\text{train}})\\).

How to read the scale — added 2026-09-26. The residual is normalised by the bare acceleration \\(\lVert a_\ell \rVert\\), which fixes three reference points:

This scale was not stated in the original version of this card, and it changes how the table below should be read. See Correction (2026-09-26) near the top.

A second, closed-form quantity — the recovered intrinsic damping \\(\gamma_{\text{geo}}\\) — is the least-squares row minimiser "the damping value that best explains this specific trajectory," independent of what \\(\gamma\\) the model was actually trained with:

$$ \gamma{\text{geo}} = -\frac{\big\langle a\ell + \Gamma(v\ell, v\ell),\ v\ell \big\rangle}{\lVert v\ell \rVert^{2}}. $$

Full derivation, practical mitigations (turning-point exclusion, reference-energy convention, integrator staggering), and validation controls (vanilla-baseline, shuffled-\\(\Gamma\\), and random-direction nulls) are in the companion note `Geodesic_Preservation_Experiment.md`.

Results: The Widest Boundary-Optimum Margin Yet

<p align="center"><img src="results/geodesicoverlayanisogaussiand1024.png" alt="PPL vs geodesic residual overlay for d=1024 aniso-Gaussian + fock-reg gamma sweep" width="660"></p> The minima coincide, at the boundary of the swept range, with the widest margin yet. Both PPL and \\(\bar{R}\\) bottom out at \\(\gamma = 0.050\\); \\(\gamma=0.05\\) beats its nearest PPL competitors by \\(\approx10\text{-}11\%\\) (25.5-26.6 PPL against \\(\gamma=0.50\\) and \\(\gamma=0.10\\) respectively) — comfortably wider than the d=768 sweep's \\(\approx4\%\\) margin and well clear of the family's informal \\(\approx5\%\\) "unreliable ranking" threshold. \\(\gamma{\text{geo}}\\) again clusters tightly: mean \\(0.9627\\), std \\(0.0030\\) — essentially flat across the range, centred about \\(0.02\\) lower than the d=768 mean (\\(0.9810\\)), a small but consistent downward drift with increasing \\(d\\) that also appears at the isotropic-Gaussian d=1024 point elsewhere in this family (\\(\gamma{\text{geo}} \approx 0.933\\)).

The per-layer residual heatmap below shows the departures from geodesic behaviour concentrating in the middle layers as \\(\gamma\\) increases, peaking sharply around layers 4-7 at \\(\gamma=0.50\\):

<p align="center"><img src="results/geodesicperlayeranisogaussian_d1024.png" alt="Per-layer geodesic residual heatmap for d=1024 aniso-Gaussian + fock-reg gamma sweep" width="660"></p> Unlike d=768, the recovered intrinsic damping here is not just flat but forms the cleanest, tightest line of any sweep in the family:

<p align="center"><img src="results/gammageorecoveryanisogaussian_d1024.png" alt="Recovered intrinsic damping vs training gamma for d=1024 aniso-Gaussian + fock-reg gamma sweep" width="660"></p>

The PPL Wiggle Without the Geodesic Wiggle

The PPL curve is not monotonic past its minimum — it rises from 244.23 (\\(\gamma=0.05\\)) to a peak at \\(\gamma=0.40\\) (298.05), then drops back down at \\(\gamma=0.50\\) (269.77) to nearly the level of \\(\gamma=0.15\\) (264.77). This is the same shape found in the d=768 sweep (a local peak in the \\(\gamma \approx 0.20\text{-}0.40\\) range, partial recovery at 0.40-0.50) — the same shape appearing at two different widths is a weak argument against pure single-run noise as the sole explanation.

What is new at this width: the geodesic-residual curve \\(\bar{R}(\gamma)\\) does not share the wiggle. \\(\bar{R}\\) rises essentially monotonically across the entire swept range (\\(1.21 \to 1.31 \to 1.48 \to 1.52 \to 1.54 \to 1.81 \to 2.04 \to 2.30\\)) — the cleanest \\(\bar{R}\\)-vs-\\(\gamma\\) curve of any sweep in this document, MLP or Gaussian, with no local peak-and-recovery pattern at all. This decouples the two diagnostics in an informative way: whatever is producing the mid-range PPL bump at \\(\gamma \approx 0.20\text{-}0.40\\) (single-seed optimisation noise, or a structural interaction between the anisotropic precision matrices and mid-range explicit friction) is not simultaneously producing a mid-range departure from geodesic behaviour — the dynamics stay increasingly non-geodesic in a smooth, monotonic way even where the loss itself wiggles. Disentangling the PPL wiggle's cause would require a 2-3 seed rerun of one or two candidates (e.g. \\(\gamma=0.30\\)-\\(0.40\\)) at one or both widths — not yet done. See `Determining_optimal_gamma_for_Fock-PARFLM.md` §14.5 for the cross-scale discussion (written before this sweep's \\(\bar{R}\\)-vs-PPL decoupling was noted).

Coincidence, Boundary, and a Second Exact Predictor Match

The two-regime closed-form damping predictor, evaluated at \\(L=16\\), \\(\bar{m}=1.4\\) using the high-\\(d\\) regime constant \\(\rho_{\text{hi}} = 0.565\\), gives the same prediction regardless of \\(d\\) within the high-\\(d\\) regime (the formula depends on \\(L\\) and \\(\bar{m}\\), not \\(d\\) directly):

$$ \gamma^{\ast}_{\text{pred}} = \frac{1.4}{16}\ln(1/0.565) = 0.0875 \times 0.571 = 0.050 $$

Empirical \\(\gamma^{\ast} = 0.050\\). Exact match, zero error. This is the second consecutive exact match at \\(L=16\\) in the high-\\(d\\) regime with the aniso-Gaussian \\(V\theta\\) (after the [d=768 sweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#coincidence-boundary-and-an-exact-predictor-match)), and the **third exact match overall** counting the original MLP-\\(V\theta\\) d=1024, L=16 result. Three independent architectures/widths, one unmodified constant (\\(\rho{\text{hi}}=0.565\\)), zero free parameters per prediction — the strongest evidence yet that \\(\rho{\text{hi}}=0.565\\) is a genuine architecture-family invariant at \\(d \gtrsim 768\\).

This also matches the winner of the original isotropic-Gaussian d=1024 partial sweep elsewhere in this family (which only completed 4/8 candidates) — giving three separate \\(V_\theta\\) variants at this width all agreeing on \\(\gamma=0.05\\).

Comparison Across All Three Widths

All three widths show coinciding PPL and geodesic-residual minima, but the pattern strengthens with scale: the margin widens (large at d=384 down to a modest 4% at d=768, then back up to a clean 10.5% at d=1024, this time without the ambiguity of an interior-bowl competing basin), and the geodesic-residual curve gets progressively cleaner — from an outlier-marred non-monotonic curve at d=384, to a genuine-but-unexplained wiggle at d=768, to a fully monotonic curve at d=1024. \\(\gamma_{\text{geo}}\\) clustering also tightens with scale in absolute spread (though drifting slightly downward in mean, \\(0.981 \to 0.963\\) from d=768 to d=1024) — consistent with a model whose intrinsic damping preference becomes both more uniform and slightly lower as width increases.

Caveats: Short-Sweep Reliability and Two Incomplete Training Logs

This is a 3,000-step, single-seed measurement per gamma, and the family has one documented case where a short-sweep ranking reversed at full training length: the d=256 TinyStories aniso-Gaussian sweep favoured \\(\gamma=0.150\\) at 3K steps, but the full 20K-step run favoured \\(\gamma=0.300\\) instead. Two considerations favour trusting this sweep's \\(\gamma=0.05\\) recommendation more than that precedent would suggest: (a) this is a boundary optimum with \\(\gamma_{\text{geo}}\\) essentially flat across the whole range — there is no competing interior basin to misrank within; and (b) the margin (~10.5%) is the widest of any boundary-optimum sweep in the family, and the predictor's independent, zero-parameter prediction lands on the exact same value.

Two data-quality notes specific to this sweep's raw artifacts:

• —Two training logs are incomplete. checkpoints/gamma_0.100/training_log.jsonl contains only the final (step-3,000) evaluation record, and checkpoints/gamma_0.200/training_log.jsonl contains only the final two (step-2,500 and step-3,000) records — both missing the earlier per-500-step entries and the step-2,000 causal_probe event that every other candidate's log contains. This looks like a Colab logging/session hiccup specific to those two candidates' runs rather than a training failure: both final checkpoints' val_ppl match sweep_summary.json and geodesic_results.json exactly, and the geodesic residual analysis (which reads directly from the checkpoint, not the training log) completed normally for both. See Causal-Leak Verification below.
• —Unlike d=768, this sweep does not show the same non-monotonic pattern in \\(\bar{R}\\) (see above) — only in PPL — so the usual "watch for the wiggle's instability signature" caveat applies specifically to the loss curve, not the geometric diagnostic, for a full run at this width.

Recommendation: \\(\gamma=0.05\\) for the d=1024, L=16 aniso-Gaussian full run — the highest-confidence recommendation of the three widths swept so far in this line.

How to Get Started

import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep"
GAMMA = "0.050"  # the recommended candidate; also available: 0.100, 0.150, 0.200, 0.250, 0.300, 0.400, 0.500

logfreq_path = hf_hub_download(repo_id=REPO, filename="results/logfreq_surprisal_openwebtext.npy")
FORCE_MAX = 2.0 / math.sqrt(1024)

config = FockMultiXiPARFConfig(
    vocab_size=50257, d=1024, max_len=1024, L=16,
    v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq", logfreq_path=logfreq_path, logfreq_init_alpha=0.1,
    init_gamma=1.0, fixed_gamma=float(GAMMA),
    causal_force=True, ln_after_step=True,
    xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
    xi_learnable=True, xi_alpha_init_mode="explicit",
    fock_version="v2", n_registers=32,
    reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
    reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
    register_repulsion=True, register_repulsion_coeff=0.05,
    prefix_causal_registers=True,
    v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16,
    v_phi_eps=0.1, v_phi_phi_hidden=128, v_phi_theta_hidden=128, v_phi_mlp_hidden=128,
    top_k=16,
    use_output_bias=True, tie_embeddings=False,
    score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3, gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True, per_layer_v_phi_scale=True,
    register_salience_decay=0.5, register_salience_threshold=0.005,
    creation_gate_hidden=64, stack_discipline=True,
    d_k=64, tau_create_init=8.0,
    per_register_tau=True, per_register_keys=True,
    ortho_register_init=True,
    force_clamp_max=FORCE_MAX,
)
model = FockMultiXiPARFLM(config)

model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=1024, K=8, n_ctx=5, n_layers=16, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(1024),
    precision_max=2.0 / 1024,
    force_norm_max=FORCE_MAX,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")            # 363,884,315
print(f"gamma={state['gamma']}  step={state['step']:,}  val_ppl={state['val_ppl']:.2f}")

Available Artifacts

Training Details

Training Data

OpenWebText, tokenized with GPT-2 BPE (vocab 50257). Each candidate trains on up to 1B tokens (early stopped at 3,000 steps, effective batch 16, block 512 -> ~24.6M tokens actually consumed) and is evaluated on a held-out 2M-token validation slice with no train/val overlap.

Training Procedure (per gamma candidate)

Causal-Leak Verification

All 8 checkpoints were trained natively with prefix_causal_registers=True from step 0. The bit-exact future-perturbation causal probe passed with max_delta=0.0 at step 2,000 for 6 of the 8 gamma candidates with an explicit training-log record. The gamma=0.100 and gamma=0.200 training logs are truncated (see caveats) and do not contain this record, but their final checkpoints match the sweep summary and geodesic-results files exactly, and the architecture and training script are identical to the other 6 candidates that did pass explicitly.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_gamma_sweep_geodesic_aniso_gaussian_fockreg_d1024.ipynb (companion repo) — self-contained Colab notebook that runs the 8-candidate sweep, the geodesic residual analysis, and produces the overlay/heatmap/recovery figures in one pass, requiring no additional training beyond the sweep itself.

Evaluation Results

OpenWebText Validation Perplexity (3,000-step sweep candidates)

These PPL values are from 3,000-step short-sweep candidates and are not comparable to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic d=384 flagship). They exist solely to rank candidate damping coefficients. Note the ranking is not simply increasing with \\(\gamma\\); see The PPL Wiggle Without the Geodesic Wiggle.

See Results: The Widest Boundary-Optimum Margin Yet for the combined PPL / geodesic-residual table.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• —Not a final trained model. Every checkpoint in this repository has been trained for only 3,000 steps — a gamma-selection diagnostic, not a language model intended for generation or downstream use. Do not compare its PPL to fully trained checkpoints elsewhere in this family.
• —Single-seed sweep. Each gamma candidate is one run; no seed-variance estimate is available. The margin between \\(\gamma=0.05\\) and its nearest competitors (~10-11%) is the widest of any boundary-optimum sweep in this family, mitigating (but not eliminating) this risk; see Caveats.
• —Two training logs are incomplete (gamma=0.100, gamma=0.200 — missing most per-step entries and the step-2,000 causal-probe record), most likely a Colab logging/session hiccup specific to those runs; the final checkpoints for both are verified consistent with the sweep-summary and geodesic-results files.
• —PPL-vs-gamma wiggle, shared with the d=768 companion sweep, is not yet explained (single-seed noise vs. a structural property of this \\(V_\theta\\) configuration) — see The PPL Wiggle Without the Geodesic Wiggle. Notably the geodesic-residual curve does not share this wiggle at this width.
• —OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering.
• —Geodesic residual is a diagnostic, not a training objective. The model was trained to minimise cross-entropy; the near-geodesic behaviour at \\(\gamma=0.05\\) is an emergent structural property, not something the loss function directly optimises for.
• —Very large \\(V_\theta\\) hypernetwork. At 252.0M parameters, \\(V\theta\\) alone is 69% of the 363.9M-parameter total — the largest \\(V\theta\\) share of any model in this family, driven by the low-rank correction \\(B_k \in \mathbb{R}^{d \times r}\\) scaling with \\(d\\).
• —No causal-leak issue. All 8 checkpoints were trained natively with prefix_causal_registers=True; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• —Hardware: 1x NVIDIA H100/A100 80GB (Google Colab)
• —Training: 8 x 3,000 steps = 24,000 total training steps across the sweep, plus inference-only geodesic residual analysis (10 validation batches x 8 checkpoints, no additional training)
• —Carbon footprint: small; a single-GPU research sweep, estimated on the order of a few kg CO2

Correction (2026-09-27) — integrator name. This model's layer step was described here as a damped Euler step. It is not. The update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which carries no velocity state at all: the velocity is the position difference h - h_prev. Undamped this is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m, i.e. Störmer–Verlet in position form. It is also not velocity-Verlet, which carries an explicit velocity through half-kick/drift/half-kick. The distinction matters because the \\(\omega \cdot dt < 2\\) stability wall is the Störmer/leapfrog bound and applies to this lineage; the semi-implicit Euler models in the same collection were never subject to it. Only the name was wrong — no measurement on this card changes.