Modelpublic

atrost/climbmix-matformer-353m-1p2b-h100-xl

sourceHugging Facecc-by-nc-4.0updated 5mo agoView on Hugging Face

0likes10downloads

Model Card

atrost/climbmix-matformer-353m-1p2b-h100-xl

Extracted XL MatFormer submodel from atrost/climbmix-matformer-353m-1p2b-h100.

Details

—Architecture: ExtractedMatFormerForCausalLM
—MatFormer size: xl
—Parameters: 352,943,360
—Source checkpoint: atrost/climbmix-matformer-353m-1p2b-h100
—Source checkpoint revision: 09960b1419b0340d8e918107e3915c0dccd852de
—Source training actual tokens: 1,201,668,096
—Slice spec: {'num_attention_heads': 16, 'num_key_value_heads': 8, 'intermediate_size': 3584}

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "atrost/climbmix-matformer-353m-1p2b-h100-xl"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype="auto",
)

trust_remote_code=True is required for the custom MatFormer extracted architecture.