PrimeIntellect/INTELLECT-1-step-15500

1---2license: apache-2.03datasets:4- PrimeIntellect/fineweb-edu5- PrimeIntellect/fineweb6- PrimeIntellect/StackV1-popular7- mlfoundations/dclm-baseline-1.0-parquet8- open-web-math/open-web-math9language:10- en11pipeline_tag: text-generation12---13 14# INTELLECT-1-step-1700015 16This is an intermediate checkpoint of INTELLECT-1. You can find the [final version](https://huggingface.co/PrimeIntellect/INTELLECT-1) as well as the [instruct one](https://huggingface.co/PrimeIntellect/INTELLECT-1-Instruct)17 18|  | Step | Model URL |19|---|------|-----------|20| -> | 17000 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-17000 |21| | 28600 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-28600 |22| | 39200 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-39200 |23| | 49200 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-49200 |24| | 59200 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-59200 |25| | 69200 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-69200 |26| | 78000 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-78000 |27| | 88000 | https://huggingface.co/PrimeIntellect/INTELLECT-1-step-88000 |28 29## **Model Overview**30**INTELLECT-1** is the first collaboratively trained 10 billion parameter language model trained from scratch on 1 trillion tokens of English text and code.31 32**INTELLECT-1** was trained on up to 14 concurrent nodes distributed across 3 continents, with contributions from 30 independent community contributors providing compute.33The training code utilizes the [prime framework](https://github.com/PrimeIntellect-ai/prime), a scalable distributed training framework designed for fault-tolerant, dynamically scaling, high-perfomance training on unreliable, globally distributed workers.34The key abstraction that allows dynamic scaling is the `ElasticDeviceMesh` which manages dynamic global process groups for fault-tolerant communication across the internet and local process groups for communication within a node35The global all-reduce was done with custom int8 all-reduce kernels to reduce the communication payload required, greatly reducing the communication overhead.36 37For more detailed technical insights, please refer to our [technical paper](https://github.com/PrimeIntellect-ai/prime).38 39## **Model Details**40- **Model Contributors**: samsja, Prime Intellect, Arcee AI, kotaro, skre_0, marlo, rodeo, Herb, Olas, superchillen, Hugging Face, mev_pete, 0xfr_, dj, primeprimeint1234, Marco Giglio, realtek, Hyperbolic, hecataeus, NWO, Virtual Machine, droll, SemiAnalysis, _waiting__, toptickcrypto, sto, Johannes, washout_segment_0b, klee41- **Release Date**: 29 Nov 202442- **Model License**: Apache 2.043 44## **Technical Specifications**45| **Parameter**       | **Value**              |46|----------------------|------------------------|47| Parameter Size       | 10B |48| Number of Layers     | 42 |49| Number of Attention Heads | 32 |50| Hidden Size          | 4096 |51| Context Length       | 8192 |52| Vocabulary Size      | 128256 |53 54## **Citations**55If you use this model in your research, please cite it as follows:56```57@article{}58```