legesher/language-decoded-community

Language Decoded — Community Code

Natively-authored multilingual code for the Language Decoded project (part of Cohere's Tiny Aya Expedition). This dataset contains code written by developers in non-English programming languages and code with significant CJK content — not mechanically transpiled or LLM-translated from English.

Experiment and proposed paper title: Language Decoded: Exploring the Impact of Native Code on Multilingual Models

This data serves as the corpus for Condition 3 ("Mixed Native Sources") and is intended to serve as the corpus for Condition 4 ("Community-Contributed Native Code") in the Language Decoded experimental ladder. See legesher/language-decoded-experiments for the canonical project description.

How Condition 3 and Condition 4 differ

Both conditions deal with native-language code, but they ask different questions:

• —Condition 3 ("Mixed Native Sources") uses code pulled from real-world public-source repositories — incidentally available code that humans wrote in or with the target language. Phase 3 trained condition-3-zh-5k from data assembled here.
• —Condition 4 ("Community-Contributed Native Code")'s design goal is code whose problem-solving logic is itself native — written as if a native speaker were approaching the problem, not English code that was later translated. Currently pending sufficient direct community contributions to assemble a stable training corpus; in neither Phase 2 nor Phase 3 evaluation. Cond-5's fully-translated data served as Phase 3's practical proxy because gathering native-authored code at scale proved difficult.

If you'd like to contribute Python (or other-language) code where you approached the problem in your native target language, the contribution interface is the `legesher/legesher-native-code` HF Space — contributions there feed into the cond-4 corpus.

Available Configs

Native code for Spanish and Urdu is not yet available.

Schema

Chinese (zh) Source Breakdown

Quality Tier Distribution

File Type Distribution

Usage

from datasets import load_dataset

# Load Chinese native code
ds = load_dataset("legesher/language-decoded-community", "zh")
train = ds["train"]  # 3,137 files
val = ds["validation"]  # 349 files

# Filter by source
wenyan = train.filter(lambda x: x["source"] == "wenyan")

# Filter by quality
high_quality = train.filter(lambda x: x["quality_tier"] in ("A", "B"))

Relationship to Other Datasets

• —[legesher/language-decoded-data](https://huggingface.co/datasets/legesher/language-decoded-data): The main training data hub. Holds the per-condition training corpora (cond-1 raw English, cond-2 Legesher-transpiled, cond-3 mixed native sources, cond-4 native-authored, cond-5 fully translated via c4ai-aya-expanse-32b).
• —[legesher/language-decoded-experiments](https://huggingface.co/datasets/legesher/language-decoded-experiments): The canonical project source-of-truth — experiment tracking, evaluation results, analysis, and the full experimental ladder.
• —[legesher/language-decoded-lora](https://huggingface.co/legesher/language-decoded-lora): LoRA adapters trained on the per-condition corpora.
• —This repo stores the raw native code with full metadata. The blended and native training datasets used for fine-tuning live in language-decoded-data.

Limitations

• —Chinese only: Currently limited to Chinese-language code. Native code for Spanish and Urdu is not yet available.
• —License uncertainty: Some files (particularly from thestack) have UNKNOWN licenses. These were included because they appeared in The Stack's permissive-license subset, but individual file licenses could not always be verified.
• —Quality variation: Quality tiers are assigned heuristically based on CJK content ratio, file size, and structural indicators. Tier D files may contain minimal native-language content.
• —Non-Python files included: Unlike the Phase 3 training corpora for cond-1, cond-2, and cond-5 — which are Python-only — this dataset includes code in multiple programming languages (Python, Java, JavaScript, Wenyan, Qi, Mulan, etc.). This reflects the reality of native-language programming ecosystems and is intentional for cond-3.
• —CJK-heavy bias: Files were selected partly based on CJK character presence, which may over-represent code with Chinese comments/strings rather than code with Chinese-language syntax.
• —Native-authored ≠ scraped: Although this corpus comes closer to native-authored code than the transpiled (cond-2) or fully-translated (cond-5) corpora, the inclusion criteria are based on CJK presence and source provenance, not on whether the original author was thinking in the target language while writing the code. Cond-4 will eventually distinguish that more cleanly via direct contribution.

Citation

@misc{language-decoded-2026,
  title={Language Decoded: Exploring the Impact of Native Code on Multilingual Models},
  author={Madison Edgar and Saad Ahmed Bazaz and Tom Sherborne and Rashik Shahjahan and Khojasteh Mirza and Sarah Jawaid and Rafay Mustafa and Sohaib Ahmed Bazaz},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/legesher/language-decoded-community}
}

License

Apache 2.0

Attribution & takedown

Portions of this dataset derive from publicly available code repositories (natively-authored code scraped from public GitHub and Gitee repositories, plus community contributions collected with consent via the collection Space), collected under a best-effort license review. If you are the author of code included here and believe it was included improperly, or you would like attribution added or your code removed:

• —open a discussion on this repository's Community tab, or
• —email Madi Edgar at support@legesher.com (the maintainer listed in CITATION.cff).

We commit to citing, and on request from authors removing, any source identified as improperly included. Removals are propagated in a new dataset revision; prior revisions remain in git history unless a takedown requires otherwise.