hreyulog/arkts-code-docstring-pro

ArkTS-CodeSearch-Pro: A Cleaned and License-Aware ArkTS Dataset

Original Dataset | Pro Dataset | Paper | Evaluation Code | Dataset Processing Code | Fine-Tuned Model

This dataset is an improved version of ArkTS-CodeSearch. It contains function-level information from ArkTS (.ets) projects, including function source code, docstrings, AST representations, and repository metadata. It is intended for code retrieval, code understanding, and AST-based research.

Compared with the original dataset, this Pro version provides the following improvements:

• —License-aware collection: only repositories with locally identified open-source licenses are included. The supported license families include Apache-2.0, MIT, BSD, GPL, LGPL, and MPL.
• —Repository quality filtering: only repositories with more than zero stars are retained.
• —Comment normalization: comment delimiters such as //, /*, */, and leading * are removed from the docstrings.
• —Noise removal: copyright/license headers, code-only comments, and cleaned docstrings shorter than five characters are excluded.
• —Stronger deduplication: complete function texts and cleaned queries are globally unique.
• —Repository-isolated evaluation: repo_split_balanced groups repositories by normalized nwo across GitHub, Gitee, and GitCode, preventing the same cross-platform repository name from appearing in multiple splits.
• —Stable split sizes: both configurations are approximately 80%/10%/10%.

The final dataset contains 31,164 examples, all extracted from .ets files. The sha column was removed because commit SHA metadata was unavailable for most source repositories; function_sha is retained as a function-content identifier.

Dataset Structure

The dataset provides two configurations:

• —repo_split_balanced: repository-level split with normalized nwo repository isolation. It contains 24,932 training, 3,116 validation, and 3,116 test examples.
• —random_shuffle: globally shuffled split with seed 20260714. It contains 24,931 training, 3,117 validation, and 3,116 test examples.

Features / Columns

Usage

from datasets import load_dataset

# Repository-isolated configuration
dataset = load_dataset(
    "hreyulog/arkts-code-docstring-pro",
    name="repo_split_balanced",
)

# Random-shuffle configuration
random_dataset = load_dataset(
    "hreyulog/arkts-code-docstring-pro",
    name="random_shuffle",
)

print(dataset["train"][0])
print(dataset["train"].features)

License Configuration

All included repositories were selected from locally identified open-source repositories. Users must still comply with the individual upstream repository licenses and attribution requirements. This Pro dataset does not grant a new license over the source code.

Citation

If you use this dataset in your research, please cite the following paper:

@misc{he2026arktscodesearchopensourcearktsdataset,
      title={ArkTS-CodeSearch: A Open-Source ArkTS Dataset for Code Retrieval},
      author={Yulong He and Artem Ermakov and Sergey Kovalchuk and Artem Aliev and Dmitry Shalymov},
      year={2026},
      eprint={2602.05550},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2602.05550},
}