1Covenant/Covenant-72B

Covenant-72B

Model Overview

Covenant-72B is the largest permissionless collaboratively trained language model, trained entirely from scratch at the 72 billion parameter scale on 1.1 trillion tokens of English text.

[image]

For more details, see the technical report. This is a base model. See Covenant-72B-Chat for the instruction-tuned variant.

Covenant-72B was trained with 20+ globally distributed participants coordinated via decentralized infrastructure on the Bittensor blockchain. Unlike prior collaborative training efforts that use whitelisted compute, Covenant-72B is the first to achieve this scale with fully permissionless participation. Training used the SparseLoCo communication-efficient optimizer to reduce bandwidth requirements across distributed nodes.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "1Covenant/Covenant-72B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("1Covenant/Covenant-72B")

input_text = "The theory of general relativity"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device)
output_ids = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Model Details

• —Compute Participants: 20+ independent contributors on Bittensor
• —Minimum Compute per Participant: 8×B200 or equivalent
• —Model License: Apache 2.0

Technical Specifications

Training Details:

• —Dataset: DCLM-baseline
• —Tokens: 1.1 Trillion
• —Optimizer: SparseLoCo (communication-efficient optimizer)

Performance on Benchmarks

All results are 0-shot accnorm (%) unless noted._

\*WinoGrande uses acc; MMLU uses acc.