dataunitylab/json-schema

1---2language:3- en4license:5- unknown6---7# JSON Schema Dataset8 9This dataset consists of a collection of JSON Schema documents collected from GitHub by searching using the Sourcegraph API.10 11# Step 1: Find a list of JSON Schema paths12 13The [Sourcegraph](https://sourcegraph.com/) code search API is used to find files with a .json extension and containing `{\n  "$schema": "https://json-schema.org/"`.14This is somewhat restrictive, but still manages to find a large number of schemas.15 16    pipenv run python slurp.py --outfile repos.csv17 18# Step 2: Fetch the history information for each file19 20We fetch every revision of each JSON Schema file.21Before downloading the files, we use the GitHub API to get the list of commit hashes.22The resulting data is saved to `commits.json`.23 24    pipenv run python fetch_history.py > commits.json25 26# Step 3: Download the JSON Schema files27 28This script will download each schema which comes from GitHub and save it into subfolders in the `data` directory.29 30    ./fetch_files.sh31 32# Step 4: Validate each JSON Schema33 34The following script will read each schema in the `data` directory and confirm that it is a valid JSON Schema.35A copy of all valid schemas will be placed in the `valid_data` directory.36Note that schemas are parsed as [JSON5](https://json5.org/) to be more permissive on what syntax is allowed but the final schemas are written as standard JSON.37 38    pipenv run python validate_schemas.py39 40# Step 5: Retrieve additional metadata41 42We also collect language information using [Fasttext](https://fasttext.cc/docs/en/language-identification.html) and fetch the associated license from the GitHub API.43 44    pipenv run python get_languages.py > languages.json45    pipenv run python get_licenses.py > licenses.json46 47# Step 6: Split into train, test, and validation48 49Finally data is split into training, test, and validation sets.50Schemas are always grouped together in the same set based on the GitHub organization they are from.51Schemas can also be checked for similarity so that very similar schemas are grouped together.52 53    pipenv run python train_split.py54