AICrossSim/clm-400m

Model Card for AICrossSim/clm-400m

A 60M parameter language model trained on 22 * 400M tokens from FineWeb-Edu dataset.

Model Details

aixsim-400M is a transformer-based language model with approximately 400 million parameters (embedding layer params excluded). It uses RMSNorm for normalization and is trained on the FineWeb-Edu dataset.

• —Developed by: AICrossSim
• —Funded by: ARIA
• —Model type: Transformer Language Model
• —Language(s) (NLP): English
• —Tokenizer: HuggingFaceTB/cosmo2-tokenizer
• —Repository: AICrossSim/NewComputeBench

Training Details

Experiment setup and training logs can be found at wandb run.

Usage

import transformers

model_name="AICrossSim/clm-400m"
model = transformers.AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)

lm-evaluation-harness