aisamdasu/QuickCoder-Dataset

1---2license: other3language:4  - ko5  - en6tags:7  - code8  - fill-in-the-middle9  - jsonl10  - tokenizer11  - moe12  - dense13pretty_name: QuickCoder-Dataset14size_categories:15  - 10M<n<100M16---17 18# QuickCoder-Dataset19 20This dataset repository stores upload-ready JSONL training checkpoints for code21completion and fill-in-the-middle training. Checkpoints are appended in22approximately 20 GiB units so they can also be copied to Google Drive and loaded23from Colab/H100 training jobs. New checkpoints use one JSONL file per 20 GiB24checkpoint. The long-term target is 400 GiB total mirrored to Hugging Face and25Google Drive.26 27## Current Upload Status28 29Only validation-passing checkpoints should be uploaded or used for training.30 31| Checkpoint | Size | JSONL | In-bundle duplicates | Status |32| --- | ---: | --- | ---: | --- |33| `checkpoint_20260611_104104_bundle01_20g` | 19.04 GiB | valid, single JSONL | 0 | uploaded to Hugging Face; Google Drive cloud verified by rclone SHA256/size check |34| `checkpoint_20260611_112534_bundle01_20g` | 19.11 GiB | valid, single JSONL | 0 | uploaded to Hugging Face; Google Drive cloud verified by rclone SHA256/size check |35| `checkpoint_20260611_143422_bundle01_20g` | 19.19 GiB | valid, single JSONL | 0 | uploaded to Hugging Face; Google Drive upload pending user confirmation |36 37Current Hugging Face dataset repo:38 39```text40aisamdasu/QuickCoder-Dataset41```42 43## Repository Layout44 45```text46README.md47dataset_guide/48tokenizer/49dense/50moe/51dataset/52  checkpoint_YYYYMMDD_HHMMSS_bundleNN_20g/53    dataset/54      checkpoint_YYYYMMDD_HHMMSS_bundleNN_20g.jsonl55dataset_guide/56  checkpoint_reports/57    checkpoint_YYYYMMDD_HHMMSS_bundleNN_20g/58      MANIFEST.json59      BALANCE_REPORT.md60      CHECKSUMS.txt61```62 63## Record Format64 65Each line is one UTF-8 JSON object. The main training payload is `text`.66Metadata can appear in top-level fields and in `meta`.67 68Common fields:69 70- `text`: canonical training string.71- `domain`: task family such as `code_fim` or `code_gen`.72- `difficulty`: coarse difficulty bucket.73- `meta.lang`: programming language.74- `meta.repo`, `meta.path`, `meta.license`, `meta.source`: source metadata when available.75- `meta.mode`: FIM ordering such as `psm` or `spm`.76 77FIM examples use explicit special tokens:78 79```text80<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>{middle}81```82 83Some FIM shards may use suffix-prefix-middle ordering for robustness.84 85## Source And License Notes86 87This is a mixed code dataset. Bundle 1 contains generated/unknown continuation88records and `the-stack-v2` FIM records with per-record source metadata such as89repository, path, and license where available. Consumers should filter by90`meta.license`, `meta.source`, and project policy before redistribution or91training.92 93The dataset card intentionally uses `license: other` because this repository94contains mixed-source records rather than one uniform license.95 96## Validation Contract97 98A checkpoint is upload-ready only when:99 100- JSONL parsing succeeds.101- Empty `text` count is zero.102- In-bundle duplicate count is zero.103- The checkpoint folder contains only one file:104  `dataset/<checkpoint>.jsonl`. Legacy checkpoints may contain several JSONL105  parts only until they are repackaged.106- `MANIFEST.json`, `BALANCE_REPORT.md`, and `CHECKSUMS.txt` are present under107  `dataset_guide/checkpoint_reports/<checkpoint>/`.108- Shared `tokenizer/`, `dense/`, and `moe/` documentation lives at repository109  root, outside checkpoint folders.110 111Do not train on a checkpoint marked `Not Upload Ready`.112