CoolFace
Datasetpublic

skymizer/common_starcoder

Common Starcoder dataset This dataset is generated from bigcode/starcoderdata. Total GPT2 Tokens: 4,649,163,171 Generation Process We filtered the original dataset with common language: C, Cpp, Java, Python and JSON. We removed some columns for mixing up with other dataset: "id", "max_stars_repo_path", "max_stars_repo_name" After removing the irrelevant fields, we shuffle the dataset with random seed=42. We filtered the data on "max_stars_count" > 300 and shuffle… See the full description on the dataset page: https://huggingface.co/datasets/skymizer/common_starcoder.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
1likes207downloads
Dataset Card

Common Starcoder dataset

This dataset is generated from bigcode/starcoderdata.

Total GPT2 Tokens: 4,649,163,171

Generation Process

  1. 1.We filtered the original dataset with common language: C, Cpp, Java, Python and JSON.
  2. 2.We removed some columns for mixing up with other dataset: "id", "maxstarsrepopath", "maxstarsreponame"
  3. 3.After removing the irrelevant fields, we shuffle the dataset with random seed=42.
  4. 4.We filtered the data on "maxstarscount" > 300 and shuffle again.
  5. 5.We further reduced the dataset size by select(range(currentsize, 2500_000)), However there are only 2.13M samples left.
  6. 6.Add "n_tokens" by using GPT2Tokenizer to count the tokens in the "content" field.