Smith42/ascl-code

1---2license: other3license_name: per-file-license4license_link: LICENSE5task_categories:6  - text-generation7language:8  - en9tags:10  - astronomy11  - astrophysics12  - source-code13  - ascl14pretty_name: ASCL Astronomy Source Code15---16 17# ASCL Astronomy Source Code18 19The [Astrophysics Source Code Library](https://ascl.net) (ASCL) is a curated registry of20source code used in astronomy and astrophysics research. This dataset contains source files21extracted from ASCL-listed repositories, paired with catalog metadata.22 23## Dataset Structure24 25### Manifest (`manifest.parquet`)26 27One row per ASCL catalog entry with the following fields:28 29| Field | Description |30| --- | --- |31| `ascl_id` | ASCL identifier (e.g., `[ascl:2306.019]`) |32| `title` | Software title |33| `authors` | Author list |34| `description` | Abstract / description from ASCL |35| `detail_url` | ASCL detail page URL |36| `repo_url` | GitHub/GitLab/Bitbucket URL (if found) |37| `code_site` | Project homepage URL |38| `ads_url` | ADS bibcode URL |39| `license_type` | Detected license (e.g., MIT, GPL-3.0) |40| `license_file` | Path to license file in repo |41 42### Source Code (`code/*.parquet`)43 44Stack-style source files extracted from cloned repositories (one row per file):45 46| Field | Description |47| --- | --- |48| `ascl_id` | ASCL identifier |49| `repo_url` | Source repository URL |50| `file_path` | Relative path within repo |51| `content` | File text content |52| `language` | Detected programming language (from file extension) |53| `license_type` | License detected from the repository |54| `size` | File size in bytes |55 56## Data Collection Methodology57 58### Phase 1: Catalog Scrape59 60The ASCL catalog is scraped to extract metadata for each entry: title, authors, description,61repository URLs, and ADS bibcode links. Only entries with a repository URL on GitHub, GitLab,62or Bitbucket proceed to Phase 2.63 64### Phase 2: Code Extraction65 66Each repository is shallow-cloned (`--depth 1`), its license file is detected and classified67via regex pattern matching, and all recognised source files are extracted into Parquet batches.68Language detection uses file extension mapping (Python, C, C++, Fortran, Julia, R,69MATLAB/Octave, IDL, Java, Rust, Go, JavaScript, Shell, and others).70 71## Limitations72 73- **Repository coverage**: only repos hosted on GitHub, GitLab, or Bitbucket are included;74  code distributed via tarballs, personal websites, or other non-git hosting is skipped.75- **Shallow clones only**: only the latest commit is captured — no version history.76- **Language detection is extension-based**: file extensions are mapped to languages; there is77  no content-based language classification.78- **License detection is regex-based**: licenses are identified by pattern matching against79  common license file names and text; unusual or custom licenses may be misclassified or80  reported as `Unknown`.81- **No deduplication**: if multiple ASCL entries point to the same repository, its files may82  appear more than once.83 84## Licensing85 86This is a multi-license dataset. Each row carries a `license_type` field indicating the87license detected for that repository. Individual source files retain their original licenses88as set by their authors. Catalog metadata originates from [ASCL](https://ascl.net).89 90## Usage91 92```python93from datasets import load_dataset94 95# Load catalog metadata96ds_manifest = load_dataset("Smith42/ascl-code", data_files="manifest.parquet")97 98# Load source code files99ds_code = load_dataset("Smith42/ascl-code", data_files="code/*.parquet")100 101# Filter to a specific license102mit_code = ds_code["train"].filter(lambda x: x["license_type"] == "MIT")103 104# Filter to Python files105python_code = ds_code["train"].filter(lambda x: x["language"] == "Python")106```