Argo1-OOAS/QuadOrbit

QuadOrbit-40M

QuadOrbit-40M is the public research checkpoint evaluated in the paper "QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models." It adds a small bounded complex recurrent memory branch to each Transformer block while retaining causal attention and SwiGLU layers.

This is a base next-token language model. It is not instruction tuned and it should not be presented as a production chatbot.

Model details

The uploaded weights are the renamed orbitoid_v2_stable_complex_orbit_attention_lm checkpoint. The mathematical model is unchanged. Only the public name is now QuadOrbit.

Usage

Install the dependencies:

pip install "torch>=2.4" "transformers>=5.0" "tokenizers>=0.20" "safetensors>=0.4"

Load and generate:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Argo1-OOAS/QuadOrbit-40M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)

inputs = tokenizer("The future of language models", return_tensors="pt").to(device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        use_cache=False,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

The repository contains custom model code, so loading requires trust_remote_code=True. Review modeling_quadorbit.py before loading remote code. This portable release uses the clear PyTorch recurrence and does not require Triton. Generation is faster on a CUDA GPU.

Training data

The model was trained from scratch on a 209,715,200-token mixture:

Training used 3,000 updates with 262,144 sampled tokens per update, or about 786.4 million token presentations. Because the prepared corpus contains 209.7 million tokens, examples were sampled more than once during training.

Evaluation

The full validation evaluation used 104,448 consecutive target tokens in FP32.

The difference is small and comes from one training seed. It does not establish an improvement over the Transformer at this scale. See quadorbit.pdf for the full method, smaller-model result, proof, and limitations.

Intended use

This release is intended for architecture research, reproducibility, analysis, and small-scale experimentation. Users may study the recurrence, reproduce the reported evaluation, or continue training under the license terms.

Limitations

• —It is a small base model and is not suitable for reliable factual assistance.
• —It has not been safety tuned or instruction tuned.
• —It may generate incorrect, biased, repetitive, or offensive text.
• —The context length is limited to 512 tokens.
• —The evaluation uses one corpus and one seed.
• —The portable recurrence does not implement a generation KV cache.

Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions.

Licenses

The model weights and repository code are licensed under the Apache License 2.0. The included research paper is licensed under CC BY 4.0. Dataset content is not redistributed here and remains under the terms of its original providers.

Citation

@misc{argo1ooas2026quadorbit,
  title  = {QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models},
  author = {Argo1-OOAS},
  year   = {2026},
  url    = {https://huggingface.co/Argo1-OOAS/QuadOrbit}
}