dimitarpg13/semsimula-fock-parflm-depthcond-vtheta

Fock-PARFLM v2.1 with Depth-Conditioned Multi-Context Gaussian V_theta (TinyStories)

The TinyStories-scale member of the depth-conditioned multi-context Gaussian \\(V\theta\\) family, whose flagship is the [OpenWebText-scale Fock-PARFLM](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext) (27.23 PPL, 53.4M params). This checkpoint uses the same \\(V\theta\\) design — a bounded mixture of Gaussian wells, one bank per xi-context channel, with a small learned per-layer depth code — at the smaller Fock-PARFLM v2.1 TinyStories scale (d=256, L=8), alongside the same MLP - \\(V_\theta\\) and SQ3-structured siblings.

Unlike its two Fock-PARFLM siblings, this checkpoint needed no causal-leak re-training: it was trained from step 0 with prefix_causal_registers=True and its bit-exact future-perturbation probes and honest-vs-standard PPL checks pass cleanly throughout training. See Training Details for the probe evidence.

This model is from the Semantic Simulation framework.

Table of Contents

• —When to Use This Model
• —Architecture
• —The Analytical Form of the Potential
• —How to Get Started
• —Training Details
• —Evaluation Results
• —SPLM Family Overview
• —Bias, Risks, and Limitations
• —Citation
• —Environmental Impact

When to Use This Model

Choose this variant over the MLP-based or SQ3-structured Fock-PARFLM v2.1 when a structurally bounded, per-horizon-conditioned potential is what you want to study — not when you want the best TinyStories PPL at this scale:

Bottom line: at this small scale (d=256, L=8, 16k steps), the boundedness of the Gaussian mixture costs a large amount of expressivity relative to the unbounded SQ3 and MLP potentials — a 6.63 PPL gap to the honest MLP baseline. The same potential family reaches a competitive 27.23 PPL on OpenWebText, but only once combined with a much larger scale (d=384, L=16), five xi-context channels instead of four, wider sparse routing, register repulsion, untied embeddings, and a graduated WSD schedule. This TinyStories checkpoint isolates the potential design in the small-scale regime and shows it is not, by itself, sufficient to match the unbounded alternatives there — see Evaluation Results.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps (layer index g = 0..7):
       |
       +-- K-EMA channels: xi^(m)_t = causal_ema(h, alpha_m)   [n_ctx=4 channels]
       |
       +-- Depth-conditioned multi-context V_theta (Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     V_m = -sum_k w_k^m exp(-0.5 a_k^m (h - mu_k^m)^2)  [8 wells per context]
       |     V_theta = sum_m V_m(xi_g^(m), h)               [4 contexts, 32 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange), prefix-causal (leak-free)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

The Analytical Form of the Potential

Each of the 4 xi-context channels gets its own bank of 8 diagonal-precision Gaussian wells. For context \\(m\\), well \\(k\\), the bank contributes

$$ Vm(\xi^{(m)}, h) = -\sum{k=1}^{8} wk^{(m)}(\xi^{(m)}) \exp\left(-\frac{1}{2} \sum{d=1}^{256} a{k,d}^{(m)}(\xi^{(m)}) \left(hd - \mu_{k,d}^{(m)}(\xi^{(m)})\right)^2\right) $$

where the mixture weights \\(wk^{(m)}\\) are a softmax over a linear projection of \\(\xi^{(m)}\\) (so they sum to 1 per context), and the per-dimension precisions \\(a{k,d}^{(m)}\\) are strictly positive via a softplus projection. Because every term is a negative exponential bounded in \\((-1, 0]\\), each bank's contribution is bounded in \\([-1, 0]\\) — this is the structural fix that gives the model a provably finite potential and force, in contrast to the SQ3 sibling's unbounded log-sum-exp mixture.

Depth conditioning. Rather than training 8 independent well banks (one per integration layer, which would multiply the parameter count by \\(L=8\\)), a single shared bank per context is reused across all layers. Each layer \\(g\\) instead learns a small additive depth code \\(e_g^{(m)} \in \mathbb{R}^{256}\\) that shifts the context before the bank sees it:

$$ \xig^{(m)} = \xi^{(m)} + eg^{(m)}, \qquad V\theta(\xi, h) = \sum{m=1}^{4} Vm(\xig^{(m)}, h) $$

Since \\(e_g^{(m)}\\) is constant with respect to \\(h\\), the gradient is unaffected by this additive shift of the input, and the resulting total potential over all 4 contexts is bounded in \\([-4, 0]\\).

Force (closed form):

$$ f\theta = -\nablah V\theta = \sum{m=1}^{4} \sum{k=1}^{8} wk^{(m)} ak^{(m)} \odot \left(h - \muk^{(m)}\right) \exp\left(-\frac{1}{2} ak^{(m)} \cdot \left(h - \muk^{(m)}\right)^2\right) $$

The Gaussian factor decays to zero away from each well centre, so the force magnitude is bounded (it peaks at a finite radius from each \\(\mu_k^{(m)}\\) and vanishes both at the centre and far away) — unlike SQ3's quadratic force, which grows without bound as \\(h\\) moves away from a well.

Empirically bounded, but far from the theoretical ceiling. The theoretical range of \\(V\theta\\) is \\([-4, 0]\\), but on this checkpoint the observed range over validation batches is only **[-0.624, -0.00002]** (mean -0.017, std 0.024) — the 32 wells are only weakly activated at this checkpoint. This is the flattest, most compressed landscape of any Fock-PARFLM \\(V\theta\\) variant published so far (see Evaluation Results), and plausibly part of why the PPL gap to the unbounded siblings is large at this scale: the force budget carried by \\(V\theta\\) here is tiny relative to \\(V\phi\\) and the Fock registers.

For the full derivation, boundedness proof, and the SARFGaussianVTheta frozen-anchor variant, see the companion note: `Structured_VTheta_Design_and_Theory.md` and the OpenWebText flagship's write-up of the same design at scale.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_gaussian_vtheta import DepthConditionedMultiContextGaussianVTheta, install_depth_routing

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, L=8,
    v_hidden=1024, v_depth=3,
    max_len=1024,
    fixed_gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16, d_k=64,
    tau_create_init=8.0,
    register_salience_decay=0.5,
    register_salience_threshold=0.01,
    stack_discipline=True,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,
)
model = FockMultiXiPARFLM(config)

# -- Swap in the depth-conditioned multi-context Gaussian V_theta --
model.V_theta = DepthConditionedMultiContextGaussianVTheta(d=256, K=8, n_ctx=4, n_layers=8)
install_depth_routing(model)  # wires the per-layer index into V_theta's depth-code lookup

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-depthcond-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly (32 wells: 4 contexts x 8 wells) --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                                # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)              # (1, 64, 4, 8, 256)
    print(f"Attractor centres shape: {centres.shape}")

Available Artifacts

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens.

Training Procedure

The base architecture is identical to Fock-PARFLM v2.1. The only modification is the \\(V\theta\\) replacement: a single shared `DepthConditionedMultiContextGaussianVTheta` bank (4 contexts x 8 wells) plus per-layer depth codes are constructed in place of the 3-layer MLP, before training begins from scratch. The pairwise \\(V\phi\\) (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged, and prefix_causal_registers=True is set from the start.

Causal-Leak Verification (no fix needed — trained natively leak-free)

Because this run started with prefix_causal_registers=True from step 1 (rather than being re-trained after the fact like the MLP and SQ3 siblings), the training log carries the same probes as a matter of routine certification, not remediation:

The deterministic architectural probe is bit-exact 0.0 at every checkpoint, and the honest-vs-standard PPL differences are both within ~1.5 standard errors of zero — not statistically distinguishable from no leak. No causal-leak caveat applies to the 16.33 PPL reported on this card.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_gaussian_vtheta.ipynb (companion repo) --- Colab notebook with the depth-conditioned multi-context Gaussian arm, GDrive output, checkpointing, and live progress display.

Evaluation Results

TinyStories Validation Perplexity

All PPL values in this table are honest, leak-free numbers. This model and the SQ3 sibling have nearly identical total parameter counts (both ~18.2M, dominated by their respective \\(V_\theta\\) hypernetworks) yet very different PPL — the gap here is attributable to the design trade-off of a bounded potential, not to parameter budget.

PPL is evaluated on the TinyStories validation split; see Training Data.

V_theta Landscape Statistics

This is the flattest, most compressed \\(V_\theta\\) landscape of any Fock-PARFLM variant published so far — a direct, structural consequence of the bounded Gaussian-mixture design rather than something the optimiser found on its own.

Learned Xi-Channel Decay Rates

The final learned alpha values \\([\alpha1, \ldots, \alpha4] = [0.321, 0.544, 0.854, 0.908]\\) (initialised at \\([0.25, 0.5, 0.75, 0.95]\\)) settle to a somewhat different spread than the MLP and SQ3 siblings' \\(\approx [0.11\text{--}0.14, 0.55\text{--}0.59, 0.79\text{--}0.84, 0.95\text{--}0.97]\\), suggesting the depth-conditioned Gaussian potential shifts slightly more weight onto the two longer-horizon xi channels.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• —Research checkpoint only. This model is a proof-of-concept for bounded, depth-conditioned scalar potentials in Fock-augmented architectures, not a production system.
• —TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• —English only. No multilingual capability.
• —Small scale. 18.2M parameters, 256-dim hidden states.
• —No safety training. No RLHF, DPO, or safety filtering has been applied.
• —V_phi and Fock forces still use autograd. Only the \\(V\theta\\) gradient is analytical; the \\(V\phi\\) pairwise force and Fock register forces still require torch.autograd.grad.
• —Large expressivity gap at this scale. The 6.63 PPL gap to the honest MLP baseline (16.33 vs 9.70) is substantially larger than the SQ3 sibling's 1.20 PPL gap. The bounded-potential design that lets the OpenWebText-scale flagship reach 27.23 PPL does not, by itself, transfer its advantage down to this much smaller model/training budget — see When to Use This Model.
• —Weakly activated wells. The empirical \\(V_\theta\\) range (0.624) is far below the theoretical bound (4.0), indicating the 32 wells are only lightly used at this checkpoint; this may partly explain the gap to the unbounded siblings.
• —No causal-leak issue. Unlike the MLP and SQ3 siblings, this checkpoint was trained natively with prefix_causal_registers=True and needed no re-training or PPL correction; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• —Hardware: NVIDIA A100 40GB (Google Colab)
• —Training time: ~3 hours (16,000 steps)
• —Carbon footprint: Estimated less than 2 kg CO2