ammarnasr/the-stack-swift-clean
Dataset 1: TheStack - Swift - Cleaned Description: This dataset is drawn from TheStack Corpus, an open-source code dataset with over 3TB of GitHub data covering 48 programming languages. We selected a small portion of this dataset to optimize smaller language models for Swift, a popular statically typed language. Target Language: Swift Dataset Size: Training: 900,000 files Validation: 50,000 files Test: 50,000 files Preprocessing: Selected Swift as the target language due to… See the full description on the dataset page: https://huggingface.co/datasets/ammarnasr/the-stack-swift-clean.
Dataset 1: TheStack - Swift - Cleaned
Description: This dataset is drawn from TheStack Corpus, an open-source code dataset with over 3TB of GitHub data covering 48 programming languages. We selected a small portion of this dataset to optimize smaller language models for Swift, a popular statically typed language.
Target Language: Swift
Dataset Size:
- Training: 900,000 files
- Validation: 50,000 files
- Test: 50,000 files
Preprocessing:
- Selected Swift as the target language due to its popularity on GitHub.
- Filtered out files with average line length > 100 characters, maximum line length > 1000 characters, and alphabet ratio < 25%.
- Split files into 90% training, 5% validation, and 5% test sets.
Tokenizer: Byte Pair Encoding (BPE) tokenizer with tab and whitespace tokens. GPT-2 vocabulary extended with special tokens.
Training Sequences: Sequences constructed by joining training data text to reach a context length of 2048 tokens (1024 tokens for full fine-tuning).