dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta

Fock-PARFLM v2.1 with Depth-Conditioned Anisotropic Multi-Context Gaussian V_theta (TinyStories)

The anisotropic sibling of the depth-conditioned multi-context Gaussian V_theta checkpoint. Both models share the identical Fock-PARFLM v2.1 scaffold (d=256, L=8, 4 xi-context channels, 16 Fock registers) and the identical bounded-mixture-of-Gaussian-wells design for \\(V\theta\\), differing in exactly one structural choice: this model's precision matrix per well is **diagonal + a learned rank-4 low-rank correction** (`Sigmak^{-1} = diag(ak) + Bk B_k^T`) instead of purely diagonal, giving each well a non-axis-aligned, ellipsoidal shape rather than an axis-aligned one. Training also adds a small auxiliary Fock-coupling log-barrier regulariser on the xi-channel coupling strengths, which the isotropic sibling did not use.

The effect of this one change is large: honest validation PPL drops from 16.33 (isotropic) to 9.04 (anisotropic) — a 7.29 PPL improvement, and, notably, the first time any bounded-potential Fock-PARFLM variant on TinyStories has overtaken the unbounded MLP (9.70) and SQ3 (10.90) baselines rather than trailing them. See When to Use This Model and Why the Anisotropic Correction Wins for the full comparison and the reasons behind the gap.

Like its isotropic sibling, this checkpoint needed no causal-leak re-training: it was trained from step 0 with prefix_causal_registers=True, and its bit-exact future-perturbation probes and honest-vs-standard PPL checks pass cleanly throughout training. See Training Details for the probe evidence.

This model is from the Semantic Simulation framework.

Table of Contents

• —When to Use This Model
• —Architecture
• —The Analytical Form of the Potential
• —The Fock-Coupling Regularisation Term
• —Why the Anisotropic Correction Wins
• —How to Get Started
• —Training Details
• —Evaluation Results
• —SPLM Family Overview
• —Bias, Risks, and Limitations
• —Citation
• —Environmental Impact

When to Use This Model

Choose this variant over the isotropic Gaussian, SQ3-structured, or MLP Fock-PARFLM v2.1 variants when you want the best TinyStories PPL currently published for this family while keeping a structurally bounded, interpretable potential:

Bottom line: at this same small scale (d=256, L=8, 20k steps) where the isotropic Gaussian trailed the unbounded siblings by a 6.63 PPL gap, the anisotropic correction plus the Fock-coupling regulariser not only close that gap but overtake both unbounded baselines, reaching a new family-best honest TinyStories PPL of 9.04 — 0.66 better than the MLP and 1.86 better than SQ3. This is not a free lunch: the anisotropic \\(V_\theta\\) hypernetwork is ~3x larger than the isotropic one (12.6M vs 4.2M parameters), so the comparison is not parameter-matched. See Why the Anisotropic Correction Wins for a fuller discussion of what is and is not driving the improvement.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps (layer index g = 0..7):
       |
       +-- K-EMA channels: xi^(m)_t = causal_ema(h, alpha_m)   [n_ctx=4 channels]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [4 contexts, 32 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange), prefix-causal (leak-free)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

The Analytical Form of the Potential

Each of the 4 xi-context channels gets its own bank of 8 wells, exactly as in the isotropic sibling. The only change is the precision structure: instead of a purely diagonal precision \\(ak \in \mathbb{R}^{256}\\), each well now carries an additional learned low-rank factor \\(Bk \in \mathbb{R}^{256 \times 4}\\), giving an effective inverse-covariance

$$ \Sigmak^{-1} = \mathrm{diag}(ak) + Bk Bk^\top, \qquad Bk \in \mathbb{R}^{256 \times 4}, \quad ak > 0. $$

With \\(\mathrm{diff}\k^{(m)} = h - \muk^{(m)}(\xi^{(m)})\\), context \\(m\\)'s contribution is

$$ Vm(\xi^{(m)}, h) = -\sum{k=1}^{8} wk^{(m)}(\xi^{(m)}) \exp\Big(-\frac{1}{2}\big[ak^{(m)} \cdot (\mathrm{diff}\k^{(m)})^2 + \lVert Bk^{(m)\top} \mathrm{diff}\_k^{(m)} \rVert^2\big]\Big) $$

where the first bracket term is the same axis-aligned quadratic form as the isotropic sibling, and the second, new term \\(\lVert Bk^\top \mathrm{diff} \rVert^2\\) adds curvature along the (up to) 4 directions spanned by \\(Bk\\)'s columns — turning each well from an axis-aligned hyper-ellipsoid into a general ellipsoid whose principal axes are learned rather than fixed to the coordinate frame. Setting rank \\(r=0\\) recovers the isotropic potential exactly (the low-rank term vanishes identically), so the two siblings are nested designs, not unrelated architectures.

Depth conditioning works exactly as in the isotropic sibling: a single shared bank per context is reused across all 8 layers, and each layer \\(g\\) learns a small additive depth code \\(eg^{(m)} \in \mathbb{R}^{256}\\) that shifts the context before the bank sees it, \\(\xig^{(m)} = \xi^{(m)} + eg^{(m)}\\), with \\(V\theta(\xi, h) = \sum{m=1}^{4} Vm(\xi_g^{(m)}, h)\\).

Force (closed form):

$$ f\theta = -\nablah V\theta = \sum{m=1}^{4}\sum{k=1}^{8} wk^{(m)}\big[ak^{(m)} \odot \mathrm{diff}\k^{(m)} + Bk^{(m)}(Bk^{(m)\top}\mathrm{diff}\k^{(m)})\big]\exp\Big(-\frac{1}{2}\big[ak^{(m)}\cdot(\mathrm{diff}\k^{(m)})^2 + \lVert Bk^{(m)\top}\mathrm{diff}\_k^{(m)}\rVert^2\big]\Big) $$

The first term inside the brackets, \\(ak \odot \mathrm{diff}\k\\), is the same axis-aligned pull as the isotropic sibling; the second term, \\(Bk(Bk^\top \mathrm{diff}\k)\\), adds a pull component along the directions \\(Bk\\) defines — this is what lets a single well exert direction-selective force (e.g. a strong pull along one axis combined with a weak or even negligible pull along a nearby one), something an axis-aligned diagonal precision cannot represent. The same Gaussian decay factor that bounds the isotropic force also bounds this one: since \\(ak > 0\\) (softplus) and \\(Bk B_k^\top\\) is positive semi-definite, the exponent argument is never positive, so every well's magnitude still peaks at a finite radius and vanishes both at the centre and far away.

Boundedness is unaffected by anisotropy. Because the exponential factor is still bounded in \\((0, 1]\\), each well's contribution is still bounded in \\([-1, 0]\\), each context's contribution in \\([-1, 0]\\), and the full \\(V_\theta\\) (summed over 4 contexts) in \\([-4, 0]\\) — identical to the isotropic sibling's theoretical bound. The low-rank correction changes the shape of the wells, not the range of the potential.

Empirical landscape. On this checkpoint the observed \\(V_\theta\\) range over validation batches is [-0.0102, -0.0000024] (mean -0.0000024, std 0.0001) — even smaller in raw magnitude than the isotropic sibling's [-0.624, -0.00002] range. This is a useful and slightly counter-intuitive data point: the PPL improvement over the isotropic sibling is not coming from a larger swing in the potential's value. See Why the Anisotropic Correction Wins for what is actually driving it.

For the full derivation, boundedness proof, and the base isotropic design, see the companion note: `Structured_VTheta_Design_and_Theory.md` and the isotropic sibling's analytical form section.

The Fock-Coupling Regularisation Term

This checkpoint's training loss carries one auxiliary term the isotropic sibling's does not: a log-barrier penalty on the xi-channels' coupling strengths — how strongly each of the 4 causal context channels stays coupled to its own past, as opposed to collapsing into a copy of the instantaneous hidden state.

Where \\(\alpha_k\\) comes from. Each xi-context channel is a causal, weighted exponential moving average (EMA) of the hidden state (see the MLP sibling for the base multi-channel-xi design):

$$ \xik[t] = \sum{s \le t} Wk[t,s] h[s], \qquad Wk[t,s] = \frac{\alphak^{t-s}}{Zk[t]}, \qquad Zk[t] = \sum{r \le t} \alpha_k^{t-r}, $$

where the decay \\(\alphak = \mathrm{sigmoid}(zk) \in (0,1)\\) is a learned, unconstrained parameter \\(zk\\) passed through a sigmoid. Two limits matter here: as \\(\alphak \to 0\\), the weight matrix concentrates entirely on \\(s=t\\), so \\(\xik[t] \to h[t]\\) — the channel degenerates into an instantaneous copy of the hidden state, contributing nothing that \\(V\theta\\) doesn't already see directly through \\(h\\) itself. As \\(\alpha_k \to 1\\), the weights approach an unweighted running mean over the whole causal prefix — maximal memory.

The regulariser. The auxiliary loss is a log-barrier on each channel's coupling strength:

$$ L{\text{fock}}(\theta) = -\lambda{\text{fock}} \sum{k=1}^{4} \log(\alphak + \epsilon), \qquad \lambda_{\text{fock}} = 0.005, \quad \epsilon = 10^{-6}, $$

added directly to the existing loss (next-token cross-entropy plus the \\(V_\theta\\) magnitude penalty):

$$ L{\text{total}} = L{\text{ntp}} + \lambdaV RV + L_{\text{fock}}(\theta). $$

Because \\(-\log(\alphak+\epsilon) \to +\infty\\) as \\(\alphak \to 0\\) and \\(\to 0\\) as \\(\alpha_k \to 1\\), this is a one-sided barrier: it penalises channels drifting toward the degenerate instantaneous limit and is essentially inert for channels already well-coupled to their causal history.

How it interacts with the rest of the model. The optimiser does not update \\(\alphak\\) directly — it updates the unconstrained \\(zk\\), with \\(\alphak = \mathrm{sigmoid}(zk)\\). Differentiating through the sigmoid,

$$ \frac{\partial L{\text{fock}}}{\partial zk} = -\lambda{\text{fock}} \frac{\alphak(1-\alphak)}{\alphak + \epsilon}. $$

For \\(\alphak \gg \epsilon\\) this reduces to \\(\approx -\lambda{\text{fock}}(1-\alpha_k)\\), which has two notable properties:

• —It does not vanish as \\(\alpha_k \to 0\\). It approaches a constant, \\(-\lambda{\text{fock}}\\) — a persistent push toward larger \\(zk\\) that does not weaken even for very small \\(\alphak\\). This is worth highlighting because a naive log-barrier applied directly to a sigmoid output usually *does* suffer vanishing gradients near the boundary (the sigmoid's own derivative \\(\alphak(1-\alphak)\\) shrinks to 0 there); here the barrier's own blow-up as \\(\alphak \to 0\\) exactly cancels that shrinkage, so the correction stays effective for however small \\(\alpha_k\\) drifts (until it gets close enough to \\(\epsilon = 10^{-6}\\) that the \\(+\epsilon\\) floor takes over).
• —It vanishes smoothly as \\(\alpha_k \to 1\\). Once a channel is well-coupled to its history, the term contributes essentially zero gradient, leaving the language-modelling loss free to place \\(\alpha_k\\) wherever it wants above the floor.

In effect, \\(L{\text{fock}}\\) acts as a floor that keeps all 4 xi-channels feeding genuinely distinct temporal contexts into the four anisotropic Gaussian well banks, rather than letting the optimiser quietly collapse one or more channels into a redundant copy of \\(h\\) — which would waste an entire bank's worth of \\(V\theta\\) capacity (8 wells, ~2.1M parameters) on an input carrying no information the rest of the model doesn't already have. This term only touches the xi-channel decays \\(zk\\); it does not directly touch \\(V\theta\\)'s well parameters (\\(\muk, ak, Bk, wk\\)) or any other part of the model — its effect on those is entirely indirect, mediated through the richer, more temporally-diverse \\(\xi\\) inputs the wells receive once channel collapse is prevented. The isotropic sibling has no such floor; empirically, this checkpoint's final decays \\([\alpha1,\ldots,\alpha4] = [0.418, 0.609, 0.788, 0.923]\\) are comfortably spread across the memory-horizon range with none collapsed toward 0, consistent with the barrier doing its intended job (see Learned Xi-Channel Decay Rates).

Why the Anisotropic Correction Wins

Three changes distinguish this checkpoint from the isotropic sibling, and all three plausibly contribute to the 7.29 PPL gap:

1. 1.Direction-selective force, not just axis-aligned force. The low-rank term \\(Bk(Bk^\top \mathrm{diff}\k)\\) lets a single well pull \\(h\\) toward its centre along directions that are *not* aligned with the coordinate axes of the residual stream. An axis-aligned diagonal precision can only stretch or compress each of the 256 dimensions independently; the rank-4 correction adds up to 4 additional, jointly-learned directions per well along which curvature can concentrate. Given that the empirical \\(V\theta\\) magnitude is smaller than the isotropic sibling's, this suggests the benefit comes primarily from the direction of the force (which shapes the trajectory of \\(h\\) through the integration steps) rather than its magnitude — a small but well-aimed push can matter more than a larger but axis-locked one.

1. 1.The Fock-coupling regulariser keeps all 4 xi-channels engaged. The isotropic sibling's wells were only weakly activated (std 0.024 on a [-4,0] scale) with no mechanism to prevent the xi-channel coupling strengths \\(\alphak\\) from drifting toward the degenerate, instantaneous corner. See [The Fock-Coupling Regularisation Term](#the-fock-coupling-regularisation-term) for the exact loss term and why its gradient stays effective even for small \\(\alphak\\) rather than vanishing. The final learned values, \\([\alpha1,\ldots,\alpha4] = [0.418, 0.609, 0.788, 0.923]\\), are comfortably spread and none has collapsed — consistent with all 4 context banks staying informative throughout training.

1. 1.A substantially larger \\(V_\theta\\) hypernetwork. The low-rank factor \\(Bk\\) is itself produced by a learned projection (`Bproj, mapping the 256-dim context into K x d x rank = 8192` numbers per bank), which adds roughly 8.4M parameters across the 4 banks. Total \\(V_\theta\\) parameters rise from 4.2M (isotropic) to 12.6M (anisotropic), and total model parameters from 18.2M to 26.6M. This comparison is therefore not parameter-matched against the isotropic sibling, the SQ3 sibling, or the MLP baseline — part of the gap may simply reflect the larger capacity available to shape the potential, independent of the anisotropy itself. Disentangling "extra capacity" from "anisotropic shape" (e.g. by comparing against an isotropic model with a similarly enlarged hypernetwork) is a natural follow-up ablation not yet run.

A damping ablation run alongside this checkpoint (gamma=0.15, otherwise identical) reached only 10.05 PPL — worse than gamma=0.30's 9.04 — suggesting the larger, more expressive anisotropic \\(V_\theta\\) benefits from the stronger damping's stabilising effect on training, consistent with the broader gamma-sweep findings reported elsewhere in the Semantic Simulation companion notes.

\\(V_\theta\\)'s shape matters more than the Fock/exchange mechanism. A separate, orthogonal comparison in the family reinforces the point above. Fock Attention with MLP V_theta pairs the same MLP \\(V\theta\\) as the original [Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) with a direct, \\(O(T^2)\\) attention-like exchange force instead of the persistent register pool, and lands at 9.42 PPL — essentially the same as the MLP-plus-register variant (9.70) and nowhere near this checkpoint's 9.04. Swapping *which* Fock/exchange mechanism is used (registers vs. direct exchange), with \\(V\theta\\) held at MLP, moves PPL by only \\(\approx\\)0.3 points; swapping \\(V\theta\\)'s shape (MLP vs. anisotropic Gaussian), with the register mechanism held fixed, moves it by \\(\approx\\)0.7 points in this checkpoint's favour and by over 7 points across the full isotropic-to-anisotropic range reported above. In other words: on TinyStories, this checkpoint's advantage over Fock Attention is best attributed to \\(V\theta\\)'s bounded, ellipsoidal-well shape, not to using registers instead of attention-style exchange. Whether that ordering holds at OpenWebText scale is still open — the anisotropic-Gaussian \\(V\theta\\) has been scaled up to OpenWebText (see the [gamma-sweep family](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep)), but no MLP-\\(V\theta\\) OpenWebText-scale checkpoint currently exists to test whether the same gap reproduces at that scale.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, max_len=1024,
    L=8, v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    logfreq_init_alpha=0.1,
    init_gamma=1.0,
    fixed_gamma=0.30,
    causal_force=True,
    ln_after_step=True,
    xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True,
    xi_alpha_init_mode="explicit",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16,
    register_salience_decay=0.5,
    register_salience_threshold=0.005,
    creation_gate_hidden=64,
    stack_discipline=True,
    d_k=64,
    tau_create_init=8.0,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,
)
model = FockMultiXiPARFLM(config)

# -- Swap in the depth-conditioned anisotropic multi-context Gaussian V_theta --
import math
model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=256, K=8, n_ctx=4, n_layers=8, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(256),
    precision_max=2.0 / 256,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)  # wires the per-layer index into V_theta's depth-code lookup

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly (32 wells: 4 contexts x 8 wells) --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                                # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)              # (1, 64, 4, 8, 256)
    print(f"Attractor centres shape: {centres.shape}")

Available Artifacts

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens.

Training Procedure

The base architecture is identical to Fock-PARFLM v2.1 and to the isotropic Gaussian sibling. The only structural modification is the \\(V\theta\\) precision: a single shared `AnisotropicDepthConditionedGaussianVTheta` bank (4 contexts x 8 wells, rank-4 low-rank correction per well) plus per-layer depth codes are constructed in place of the isotropic bank, before training begins from scratch. The pairwise \\(V\phi\\) (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged, and prefix_causal_registers=True is set from the start. Training additionally applies the Fock-coupling log-barrier regulariser described in The Fock-Coupling Regularisation Term (lambda=0.005, eps=1e-6), which the isotropic sibling did not use.

Causal-Leak Verification (no fix needed — trained natively leak-free)

Because this run started with prefix_causal_registers=True from step 1 (rather than being re-trained after the fact like the MLP and SQ3 siblings), the training log carries the same probes as a matter of routine certification, not remediation:

The deterministic architectural probe is bit-exact 0.0 at every checkpoint, and the honest-vs-standard PPL differences are both within ~2 standard errors of zero — not statistically distinguishable from no leak. No causal-leak caveat applies to the 9.04 PPL reported on this card.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_aniso_gaussian_fockreg_tinystories.ipynb (companion repo) --- Colab notebook with the depth-conditioned anisotropic Gaussian arm, GDrive output, checkpointing, and live progress display. A gamma=0.15 ablation was run from the same notebook; see Why the Anisotropic Correction Wins.

Evaluation Results

TinyStories Validation Perplexity

All PPL values in this table are honest, leak-free numbers. This model is the first bounded-\\(V\theta\\) Fock-PARFLM variant on TinyStories to beat the unbounded Matched-Attention baseline's near-neighbours (MLP, SQ3) rather than trail them — at the cost of a larger \\(V\theta\\) hypernetwork (26.6M vs 17.4-18.2M total parameters); see Why the Anisotropic Correction Wins.

PPL is evaluated on the TinyStories validation split; see Training Data.

V_theta Landscape Statistics

Despite having the best PPL of any Fock-PARFLM variant on TinyStories, this model has the flattest \\(V\theta\\) landscape ever observed in this family — a reminder that raw potential magnitude is not a reliable proxy for how much useful work \\(V\theta\\) is doing; see Why the Anisotropic Correction Wins.

Learned Xi-Channel Decay Rates

The final learned alpha values \\([\alpha1, \alpha2, \alpha3, \alpha4] = [0.418, 0.609, 0.788, 0.923]\\) (initialised at \\([0.25, 0.5, 0.75, 0.95]\\)) are the most evenly spread of any Fock-PARFLM sibling published so far — consistent with the Fock-coupling regulariser doing its job of keeping every context channel engaged rather than letting the optimiser collapse one or more of them.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• —Research checkpoint only. This model is a proof-of-concept for bounded, anisotropic, depth-conditioned scalar potentials in Fock-augmented architectures, not a production system.
• —TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• —English only. No multilingual capability.
• —Not parameter-matched against its siblings. At 26.6M total parameters (12.6M in \\(V\theta\\) alone), this model is ~45% larger than the isotropic Gaussian, SQ3, and MLP siblings (17.4-18.2M each). Part of its PPL advantage may be attributable to the larger \\(V\theta\\) hypernetwork rather than to anisotropy per se; a parameter-matched ablation has not yet been run. See Why the Anisotropic Correction Wins.
• —No safety training. No RLHF, DPO, or safety filtering has been applied.
• —V_phi and Fock forces still use autograd. Only the \\(V\theta\\) gradient is analytical; the \\(V\phi\\) pairwise force and Fock register forces still require torch.autograd.grad.
• —Extremely flat empirical landscape. The empirical \\(V\theta\\) range (0.0102) is far below the theoretical bound (4.0) and even smaller than the already-flat isotropic sibling's — despite this being the best-performing \\(V\theta\\) variant published so far. This counter-intuitive combination (best PPL, flattest landscape) is discussed in Why the Anisotropic Correction Wins; it should not be read as evidence that the wells are unused, only that their magnitude is not what matters.
• —Fock-coupling regulariser is an added training-time term. The auxiliary log-barrier loss on \\(\alpha_k\\) is not present in the isotropic sibling's training, which is a second confound alongside parameter count when attributing the PPL gap to "anisotropy" specifically.
• —No causal-leak issue. Like the isotropic sibling, this checkpoint was trained natively with prefix_causal_registers=True and needed no re-training or PPL correction; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• —Hardware: NVIDIA A100 40GB (Google Colab)
• —Training time: ~4 hours (20,000 steps)
• —Carbon footprint: Estimated less than 3 kg CO2