david-thrower/smollm-corpus-instruct-2M-cosmopedia-v2-gpt2-v2-streaming

A corpus of high quality fine tuning data meant for fine tuning various HelixLM models

Dataset Composition:

• —A subset sampled from randomly selected shards from https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus cosmopedia-v2 split ...
• —Added: A preprocessed column formattedconversation concatenating the prompt, text and special tokens for instruct fine tuning.
• —Added: A token count column tokencount based on GPT2 tokenizer + the fine tuning special tokens.

Dataset's Purpose

Dataset was designed to pretrain https://github.com/david-thrower/HelixLM 's 41M parameter model proof of concept. HelixLM is a novel model architecture that out-punches a weight class above its own by combining:

• —Our novel innovation in model topology https://github.com/david-thrower/cerebros-core-algorithm-alpha which emulates the neuroscience concept of neural columns and modularity. Modularity is a concept where local clusters of densely linked neurons specialize in various aspects of a problem being solved, while the sparse connectivity between neural columns optimizes efficient communication between them and fusion of the fragments of the problem to one cohesive solution.
• —+ Linking heterogenous linear and optionally full attention layers in this connectivity pattern.
• —+ Recurrent Depth (LTI + ACT)
• —+ Modern Primitives like RoPE, SwiGLU, RMSNorm

<!-- ml-intern-provenance -->

Generated by ML Intern

This dataset repository was generated with the assistance of ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• —Try ML Intern: https://smolagents-ml-intern.hf.space
• —Source code: https://github.com/huggingface/ml-intern