lekdan/osu-everything-tools
osu-everything osu-everything contains the source tools and runbooks for maintaining the compact v1 osu! dataset in: hf://buckets/lekdan/osu-everything The source checkout is a normal Git repo. Runtime dataset state lives outside the checkout in a sibling dataset workspace. ~/osu_model/ # source checkout: code, docs, schemas, feature pipeline ~/osu_model_dataset/ # working dataset state: archives, data, fetcher DB, logs ~/osu_model_features/ # derived feature store… See the full description on the dataset page: https://huggingface.co/datasets/lekdan/osu-everything-tools.
osu-everything
osu-everything contains the source tools and runbooks for maintaining the compact v1 osu! dataset in:
hf://buckets/lekdan/osu-everythingThe source checkout is a normal Git repo. Runtime dataset state lives outside the checkout in a sibling dataset workspace.
~/osu_model/ # source checkout: code, docs, schemas, feature pipeline
~/osu_model_dataset/ # working dataset state: archives, data, fetcher DB, logs
~/osu_model_features/ # derived feature store outputThe scripts default to that sibling layout. Override it with DATASET_ROOT=/path/to/workspace when needed.
Bucket Layout
osu-everything/
├── archives/sha256/aa/bb/<archive_sha256>.osz
├── data/v1/
│ ├── all_revisions/<table>/part-<batch>.parquet
│ └── latest/<table>/part-*.parquet
├── schemas/v1/*.schema.json
├── features/v1/
│ ├── schemas/
│ │ ├── features_v1.schema.json
│ │ ├── feature_tables/*.schema.json
│ │ └── state_tables/*.schema.json
│ ├── docs/
│ ├── state/
│ ├── tables/
│ └── audio/
└── state/fetcher/state.dbThe bucket stores raw .osz archives and compact Parquet metadata. It does not publish extracted blobs, manifests, train/test splits, or a global blob registry.
schemas/v1/ is only for the compact dataset contract. Derived feature schemas live under features/v1/schemas/; do not create schemas/v1/features.json or put feature table schemas beside the compact dataset schemas.
all_revisions is the append-only metadata history. latest is regenerated from that history for quick reads and ML workloads.
features/v1/ is the derived ML feature store. Its schema and docs are stored inside the feature subtree so a bucket sync of features/v1 carries both the Parquet files and the feature data contract.
Loading Data
Sync only the prefix you need:
hf sync \
hf://buckets/lekdan/osu-everything/data/v1/latest/beatmaps \
./osu-everything/data/v1/latest/beatmapsLoad with PyArrow:
import pyarrow.dataset as ds
beatmaps = ds.dataset(
"./osu-everything/data/v1/latest/beatmaps",
format="parquet",
).to_table()Raw audio, images, videos, and .osu text remain inside the original .osz archives under archives/. Metadata rows carry archive paths and member paths needed to locate those bytes.
Derived feature rows can be loaded the same way:
import pyarrow.dataset as ds
objects = ds.dataset(
"./osu-everything/features/v1/tables/object_tokens_osu",
format="parquet",
).to_table()Updating
Clone or update source with Git:
git clone https://huggingface.co/datasets/lekdan/osu-everything-tools ~/osu_model
cd ~/osu_model
git pull --ff-onlyBuild the tools:
uv sync
cargo build --release -p osu_indexer -p osu_fetcherConfigure local osu! credentials in the dataset workspace:
mkdir -p ~/osu_model_dataset
cp osu_fetcher.toml.example ~/osu_model_dataset/osu_fetcher.toml
# fill in ~/osu_model_dataset/osu_fetcher.tomlRun a normal update:
UPLOAD=1 DOWNLOAD_CONCURRENCY=128 bash scripts/update_maps_v1.shThe wrapper hydrates only bucket data/, schemas/, and state/fetcher/, fetches new or changed .osz files, ingests them, compacts metadata by default, regenerates latest views, and publishes the changed dataset prefixes. It does not download the historical archive corpus.
See docs/PIPELINE_RUNBOOK.md for operational details.
Derived ML features are built separately after compact metadata has been updated. See osu_feature_pipeline/README.md.
Source Changes
Make code and documentation changes in ~/osu_model, then use normal Git:
git status --short
git add README.md docs scripts python crates schemas osu_feature_pipeline Cargo.toml Cargo.lock pyproject.toml uv.lock
git commit -m "describe the source change"
git pushDo not put runtime dataset state in the source commit. The source repo ignores archives/, data/, .fetcher/, .scratch/, incoming_osz/, logs/, target/, .venv/, and local credential files.
Licensing
Beatmaps frequently include copyrighted songs, backgrounds, videos, and storyboard assets. Do not publish this dataset publicly without the appropriate rights. Refer to the osu! beatmap licence before sharing.
