BosonicJustin/transcendent-logic-model

<!-- RELEASE NOTE: No model-weight license has been assigned; publication does not imply a license grant. --> <!-- RELEASE NOTE: MBPP evaluation is pending. -->

Transcendent Logic Model

Transcendent Logic Model is a 1,283,557,376-parameter decoder-only base language model trained from scratch for code completion and general text continuation. It is Llama-architecture compatible, but it does not contain or derive from Meta Llama weights. This is a pretrained base model, not an instruction-tuned chat model.

Release status

The training values above were copied from the accepted final checkpoint and structured training log. The revision is the immutable initial weight-upload commit.

Model architecture

The model uses grouped-query causal self-attention, SwiGLU feed-forward blocks, pre-normalization, and no linear biases. The Hugging Face artifact uses the standard LlamaForCausalLM implementation and does not require remote code.

Tokenizer and prompt format

The tokenizer is bigcode/starcoder2-tokenizer at revision 9cfe60e28fd01cc1391ecd2146a34cda7534efeb. Its vocabulary has 49,152 entries. The shared end-of-text/BOS/EOS/unknown token has ID 0; there is no dedicated padding token. Pretraining does not insert BOS at document starts, and the release does not define a chat template.

Use completion-style prompts and do not prepend a chat or instruction wrapper:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BosonicJustin/transcendent-logic-model"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=False,
)

prompt = "def fibonacci(n):\n    \"\"\"Return the nth Fibonacci number.\"\"\"\n"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.inference_mode():
    generated = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=128,
        eos_token_id=0,
        pad_token_id=0,
    )
print(
    tokenizer.decode(
        generated[0, inputs.input_ids.shape[1]:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )
)

Always set max_new_tokens; prompt plus completion must not exceed 4,096 tokens. For variable-length batches, alias padding to EOS at runtime without adding a vocabulary entry, use left padding, and pass the tokenizer-produced attention mask:

tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
inputs = tokenizer(
    prompts,
    padding=True,
    return_tensors="pt",
    add_special_tokens=False,
)

Pretraining data

The frozen run-1 order contains 12,836,736 unique packed rows and 52,579,270,656 input positions, with no selected row repeated. The selected mixture is 40% Python, 40% other programming languages, and 20% English by input positions.

The source terms above describe dataset metadata and do not by themselves select a license for the released model weights. Users are responsible for reviewing applicable source terms and the model-weight license selected by the publisher.

Code uses Stack v3's upstream file-level cross-repository MinHash/LSH deduplication. The local pipeline additionally applies global byte-exact and normalized-identical canonicalization, basic quality filters, benchmark guards, and leakage-safe source/repository grouping before train/validation/test split assignment. It does not apply a second fuzzy or semantic near-deduplication pass to code or English, so semantic duplicates may remain.

Documents are packed into fixed 4,096-token rows. During native pretraining, attention is block-diagonal and causal within each document, positions reset at document boundaries, and cross-document next-token labels are ignored. Ordinary Hugging Face inference remains correct when each prompt occupies its own batch row. Do not concatenate unrelated examples into one row without supplying an equivalent block-diagonal attention mechanism.

Training procedure

The run-1 training plan uses six H100 SXM 80 GB GPUs, BF16 forward/backward computation with FP32 parameters, replicated PyTorch DDP, AdamW, an effective batch of 192 rows (786,432 input positions), and 66,858 optimizer updates. The learning rate warms up to 3e-4 over 1,000 updates and decays to 3e-5. Validation uses an immutable held-out order every 500 updates; checkpoints are written every 1,000 updates.

The accepted final checkpoint completed the intended 52.58-billion-token trajectory.

Evaluation

MBPP is held out from training and checkpoint selection. Acquisition and curation use exact/normalized benchmark fingerprints and source/path guards; these controls reduce direct leakage but cannot prove the absence of every semantic paraphrase.

Evaluation artifact or immutable report: pending.

Generated code must be executed only in an isolated sandbox with network access disabled and strict time, memory, and process limits.

Intended use and limitations

The intended use is research on small code-language models, completion, and controlled post-training experiments. The model is not instruction tuned and may ignore natural-language requests, continue prompts instead of answering them, produce syntactically invalid or insecure code, reproduce undesirable training-data patterns, or make confident factual errors. Generated code must be reviewed and tested before use. This release is not suitable for autonomous deployment or security-sensitive decisions.

Reproducibility and integrity

Weights are exported from the trusted native checkpoint into standard safetensors without remote code. HF_RELEASE_MANIFEST.json records SHA-256 and size for every staged release file, while the native optimizer/RNG checkpoint is retained separately and is not uploaded. The immutable initial weight-upload commit is recorded in the release-status table above.