Smith42/ascl-code

ASCL Astronomy Source Code

The Astrophysics Source Code Library (ASCL) is a curated registry of source code used in astronomy and astrophysics research. This dataset contains source files extracted from ASCL-listed repositories, paired with catalog metadata.

Dataset Structure

Manifest (manifest.parquet)

One row per ASCL catalog entry with the following fields:

Source Code (code/*.parquet)

Stack-style source files extracted from cloned repositories (one row per file):

Data Collection Methodology

Phase 1: Catalog Scrape

The ASCL catalog is scraped to extract metadata for each entry: title, authors, description, repository URLs, and ADS bibcode links. Only entries with a repository URL on GitHub, GitLab, or Bitbucket proceed to Phase 2.

Phase 2: Code Extraction

Each repository is shallow-cloned (--depth 1), its license file is detected and classified via regex pattern matching, and all recognised source files are extracted into Parquet batches. Language detection uses file extension mapping (Python, C, C++, Fortran, Julia, R, MATLAB/Octave, IDL, Java, Rust, Go, JavaScript, Shell, and others).

Limitations

• —Repository coverage: only repos hosted on GitHub, GitLab, or Bitbucket are included; code distributed via tarballs, personal websites, or other non-git hosting is skipped.
• —Shallow clones only: only the latest commit is captured — no version history.
• —Language detection is extension-based: file extensions are mapped to languages; there is no content-based language classification.
• —License detection is regex-based: licenses are identified by pattern matching against common license file names and text; unusual or custom licenses may be misclassified or reported as Unknown.
• —No deduplication: if multiple ASCL entries point to the same repository, its files may appear more than once.

Licensing

This is a multi-license dataset. Each row carries a license_type field indicating the license detected for that repository. Individual source files retain their original licenses as set by their authors. Catalog metadata originates from ASCL.

Usage

from datasets import load_dataset

# Load catalog metadata
ds_manifest = load_dataset("Smith42/ascl-code", data_files="manifest.parquet")

# Load source code files
ds_code = load_dataset("Smith42/ascl-code", data_files="code/*.parquet")

# Filter to a specific license
mit_code = ds_code["train"].filter(lambda x: x["license_type"] == "MIT")

# Filter to Python files
python_code = ds_code["train"].filter(lambda x: x["language"] == "Python")