ahmetggg/Dr-Zeon-Github-Python-Code-Dataset

Luck Spark 1B - High Quality Code Dataset

The first quality-scored, star-agnostic code dataset for training 1B MoE code models.

Unlike The Stack / CodeParrot that filter by stars, this dataset scores every file by its content (0-10). A 2-star well-documented library scores higher than a 10k-star minified file. Continuously updated by an autonomous bot.

Repo: ahmetggg/luck-spark-1b-code-dataset | Bot: github_to_hf_bot.py | License: Permissive only (MIT / Apache-2.0 / BSD / Unlicense)

Why This Dataset is Different?

Quality Score (0-10) breakdown:

• —+3 AST parse + has function/class + docstring
• —+2 Comment ratio 5-40% (documented, not spam)
• —+1 Ideal size 500-20k chars
• —+1 Diversity (unique lines >60%)
• —+1 Weak star bonus log10(stars+1)*0.5 (max 1 point)
• —- fail minified, auto-generated, binary, 0/50 diversity

• —score <5 → discarded (trash)
• —score 5-7 → kept locally, not pushed (medium)
• —score 7+ → pushed to HF (high quality only)

You can see the exact scorer: quality_score() in github_to_hf_bot.py:26

Dataset Structure

{
  "text": "import math\nclass Calculator:\n    ...",  # raw code
  "repo": "ahmetggg/example-repo",                  # source repo
  "path": "src/calc.py",                            # file path
  "language": ".py",                                # .py/.js/.rs/.go/.java/.cpp/.ts
  "hash": "a1b2c3d4e5f6g7h8",                        # SHA256 dedup
  "score": 7.4,                                     # 0-10 quality
  "stars": 12                                       # repo stars at scrape time
}

Languages: Python, JavaScript, Rust, Go, Java, C++, TypeScript (balanced, no star bias)

Usage

from datasets import load_dataset

# Load high-quality only (7+ already filtered)
ds = load_dataset("ahmetggg/luck-spark-1b-code-dataset")
print(ds)
# DatasetDict({ train: Dataset({ num_rows: 1000+, features: [...] }) })

# Filter even stricter (e.g., 8+)
high = ds["train"].filter(lambda x: x["score"] >= 8)
print(f"Elite: {len(high)} files")

# Language split
py = ds["train"].filter(lambda x: x["language"] == ".py")

# For pretraining (raw text)
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ahmetggg/luck-spark-1b")
texts = ds["train"]["text"]

For Luck Spark 1B training:

# Pretrain: use raw text
# Instruct: use text + auto-generated instruction (coming soon)
# RL: execution-verified subset (score 8+)

Stats (Live)

• —Total repos scanned: 616+ (7 languages × 3 pages, growing)
• —Files kept: ~60% (0/50 for trash repos, 34/50 for gems)
• —Avg score: 6.2 - 7.6 (pushed avg >7.0)
• —Dedup: SHA256, ~5% duplicates removed
• —Licenses: MIT / Apache-2.0 / BSD / Unlicense only (commercial safe)

Updated continuously. Last bot run: see commit history.

Collection Method

1. 1.GitHub Search API: language:python license:mit (no star filter, sort:updated)
2. 2.Tree API: max 50 files / repo, <500KB, allowed extensions
3. 3.Raw download + quality_score() -> keep 5+, push 7+
4. 4.Arrow/Parquet -> push_to_hub every 1000 files

No manual curation. Fully autonomous, reproducible.

Limitations & Ethics

• —Only permissive licenses. No GPL/copyleft. Check repo field before commercial use.
• —Code may contain biases from GitHub. Filter score for your use-case.
• —No PII scrubbing beyond GitHub public data. Report issues via Discussions.

Citation

@dataset{luck_spark_1b_2026,
  title={Luck Spark 1B High Quality Code Dataset},
  author={ahmetggg},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/ahmetggg/luck-spark-1b-code-dataset}
}

Roadmap

• —[x] Quality-scored v1 (7+ push)
• —[ ] Execution-verified subset (python -m py_compile + tests)
• —[ ] Instruction pairs (explain this code / complete this function)
• —[ ] 100B tokens target for 1B MoE pretraining

Built for Luck Spark 1B (Mamba + MoE, Executor + Architect) - open source, HF first.

Questions? Open a Discussion on HF or check `github_to_hf_bot.py` for the exact logic.