hreyulog/arkts-code-docstring-for-anonymous

ArkTS-CodeSearch-Pro: A Cleaned and License-Aware ArkTS Dataset

Evaluation Code |Dataset processing Code | Fine-Tuned Model

This dataset contains function-level information from ArkTS (.ets) projects, including function source code, normalized docstrings, AST representations, and repository metadata. It is intended for code retrieval, code understanding, and AST-based research.

Compared with the original release, this Pro version provides the following improvements:

• —License-aware collection: only repositories with locally identified open-source licenses are included. The supported license families include Apache-2.0, MIT, BSD, GPL, LGPL, and MPL.
• —Repository quality filtering: only repositories with more than zero stars are retained.
• —Comment normalization: comment delimiters such as //, /*, */, and leading * are removed from docstrings.
• —Noise removal: copyright/license headers, code-only comments, and cleaned docstrings shorter than five characters are excluded.
• —Stronger deduplication: complete function texts and cleaned queries are globally unique.
• —Repository-isolated evaluation: repo_split_balanced groups repositories by normalized nwo across GitHub, Gitee, and GitCode, preventing the same cross-platform repository name from appearing in multiple splits.
• —Stable split sizes: both configurations are approximately 80%/10%/10%.

The final dataset contains 31,164 examples, all extracted from .ets files. The sha column was removed because commit SHA metadata was unavailable for most source repositories; function_sha is retained as a function-content identifier.

Dataset Structure

The dataset provides two configurations:

• —repo_split_balanced: repository-level split with normalized nwo repository isolation. It contains 24,932 training, 3,116 validation, and 3,116 test examples.
• —random_shuffle: globally shuffled split with seed 20260714. It contains 24,931 training, 3,117 validation, and 3,116 test examples.

Features / Columns

Usage

from datasets import load_dataset

# Repository-isolated configuration
dataset = load_dataset(
    "XXX/arkts-code-docstring-for-anonymous",
    name="repo_split_balanced",
)

# Random-shuffle configuration
random_dataset = load_dataset(
    "XXX/arkts-code-docstring-for-anonymous",
    name="random_shuffle",
)

print(dataset["train"][0])
print(dataset["train"].features)

License Configuration

All included repositories were selected from locally identified open-source repositories. Users must still comply with the individual upstream repository licenses and attribution requirements. This dataset does not grant a new license over the source code.