aisamdasu/QuickCoder-Dataset
QuickCoder-Dataset This dataset repository stores upload-ready JSONL training checkpoints for code completion and fill-in-the-middle training. Checkpoints are appended in approximately 20 GiB units so they can also be copied to Google Drive and loaded from Colab/H100 training jobs. New checkpoints use one JSONL file per 20 GiB checkpoint. The long-term target is 400 GiB total mirrored to Hugging Face and Google Drive. Current Upload Status Only validation-passing… See the full description on the dataset page: https://huggingface.co/datasets/aisamdasu/QuickCoder-Dataset.
140
1# MoE Architecture2 3The MoE model is the main scaling direction for code completion. The strongest4current structure is sparse FFN experts with shared attention and optional5multi-token prediction heads.6 7## Recommended Structure8 9- Decoder-only Transformer.10- RMSNorm before attention and FFN.11- RoPE positional encoding.12- Grouped-query attention.13- SwiGLU experts.14- Top-2 routing during training.15- Shared expert for common syntax and indentation.16- Router z-loss and auxiliary load-balancing loss.17- QK norm for long-context attention stability.18- Expert-choice telemetry from the first step, not after loss diverges.19- Optional multi-token prediction heads with low loss weight.20 21## Best Current Shape22 23Use shared attention and sparse FFN. Keep early layers mostly dense/shared so24the model learns lexical syntax before specializing.25 26- Early blocks: dense FFN or shared expert heavy.27- Middle blocks: sparse MoE every other block.28- Late blocks: sparse MoE with stricter router stability checks.29- Expert count: start with 8 experts for smoke, then 16-32 for H100 ablation.30- Routing: top-2 with capacity factor 1.25 during training.31- Inference ablation: top-1 only after top-2 training is stable.32 33For code completion, the highest-value specialization is not one expert per34language. Better targets are syntax/indentation, FIM reconstruction,35repository-context patterns, generated-code continuation, and real-code APIs.36 37## Routing Metrics38 39Log:40 41- Per-layer expert load.42- Per-expert token fraction.43- Dropped-token count.44- Router entropy.45- Auxiliary loss.46- Domain-to-expert correlation.47 48## Failure Modes49 50- Expert collapse.51- Router churn.52- Synthetic Python overfitting.53- MTP loss overpowering next-token loss.54- Shared expert absorbing too much load.55- Dropped tokens above 1% for sustained windows.56