aisamdasu/QuickCoder-Dataset
QuickCoder-Dataset This dataset repository stores upload-ready JSONL training checkpoints for code completion and fill-in-the-middle training. Checkpoints are appended in approximately 20 GiB units so they can also be copied to Google Drive and loaded from Colab/H100 training jobs. New checkpoints use one JSONL file per 20 GiB checkpoint. The long-term target is 400 GiB total mirrored to Hugging Face and Google Drive. Current Upload Status Only validation-passing… See the full description on the dataset page: https://huggingface.co/datasets/aisamdasu/QuickCoder-Dataset.
140
1# MoE Training Plan2 3## Data Mix4 5Train only on manifest-verified JSONL checkpoints. Do not point the trainer at6raw source directories.7 8Primary input:9 10```text11dataset/<checkpoint>/dataset/*.jsonl12```13 14Each 20 GiB checkpoint should be treated as an atomic training unit. Keep the15checkpoint report beside the run metadata so loss curves can be traced back to16the exact JSONL files.17 18## Curriculum19 201. Warm up on short and medium FIM examples.212. Add long repository-context examples.223. Add code generation as an auxiliary continuation task.234. Fine-tune with real-code FIM weighted higher than synthetic continuation.24 25## H100 Defaults26 27- Context: start at 2048, then ablate 4096.28- Precision: bf16 on H100.29- Optimizer: AdamW.30- Grad clipping: enabled.31- MoE aux loss: start small and tune from expert load.32- Top-k: top-2 for training, top-1 as inference ablation only.33- Capacity factor: 1.25 for training.34- Router z-loss: enabled from step 0.35- Router jitter: small nonzero value during early training.36- MTP loss: optional, low coefficient, disabled if next-token validation37 worsens.38 39## Preprocessing40 41- Preserve whitespace and FIM markers exactly.42- Append `<|endoftext|>` between packed records.43- Bucket by token length before batching.44- Track domain and language metadata for router analysis.45- Do not rebalance by duplicating JSONL files. Rebalance in the dataloader or46 sampler to avoid SSD growth.47 48## Router Guardrails49 50- If one expert holds more than 35% sustained load, raise aux loss or increase51 shared capacity.52- If dropped tokens exceed 1%, reduce batch tokens or increase capacity.53- If router entropy collapses early, lower LR or increase warmup.54- If domain-to-expert correlation is too strong, improve data mix before adding55 experts.56