CoolFace
Datasetpublic

lekdan/osu-everything-tools

osu-everything osu-everything contains the source tools and runbooks for maintaining the compact v1 osu! dataset in: hf://buckets/lekdan/osu-everything The source checkout is a normal Git repo. Runtime dataset state lives outside the checkout in a sibling dataset workspace. ~/osu_model/ # source checkout: code, docs, schemas, feature pipeline ~/osu_model_dataset/ # working dataset state: archives, data, fetcher DB, logs ~/osu_model_features/ # derived feature store… See the full description on the dataset page: https://huggingface.co/datasets/lekdan/osu-everything-tools.

sourceHugging Faceotherupdated 5mo agoView on Hugging Face
0likes151downloads
Dataset Card

osu-everything

osu-everything contains the source tools and runbooks for maintaining the compact v1 osu! dataset in:

text
hf://buckets/lekdan/osu-everything

The source checkout is a normal Git repo. Runtime dataset state lives outside the checkout in a sibling dataset workspace.

text
~/osu_model/          # source checkout: code, docs, schemas, feature pipeline
~/osu_model_dataset/  # working dataset state: archives, data, fetcher DB, logs
~/osu_model_features/ # derived feature store output

The scripts default to that sibling layout. Override it with DATASET_ROOT=/path/to/workspace when needed.

Bucket Layout

text
osu-everything/
├── archives/sha256/aa/bb/<archive_sha256>.osz
├── data/v1/
│   ├── all_revisions/<table>/part-<batch>.parquet
│   └── latest/<table>/part-*.parquet
├── schemas/v1/*.schema.json
├── features/v1/
│   ├── schemas/
│   │   ├── features_v1.schema.json
│   │   ├── feature_tables/*.schema.json
│   │   └── state_tables/*.schema.json
│   ├── docs/
│   ├── state/
│   ├── tables/
│   └── audio/
└── state/fetcher/state.db

The bucket stores raw .osz archives and compact Parquet metadata. It does not publish extracted blobs, manifests, train/test splits, or a global blob registry.

schemas/v1/ is only for the compact dataset contract. Derived feature schemas live under features/v1/schemas/; do not create schemas/v1/features.json or put feature table schemas beside the compact dataset schemas.

all_revisions is the append-only metadata history. latest is regenerated from that history for quick reads and ML workloads.

features/v1/ is the derived ML feature store. Its schema and docs are stored inside the feature subtree so a bucket sync of features/v1 carries both the Parquet files and the feature data contract.

Loading Data

Sync only the prefix you need:

bash
hf sync \
  hf://buckets/lekdan/osu-everything/data/v1/latest/beatmaps \
  ./osu-everything/data/v1/latest/beatmaps

Load with PyArrow:

python
import pyarrow.dataset as ds

beatmaps = ds.dataset(
    "./osu-everything/data/v1/latest/beatmaps",
    format="parquet",
).to_table()

Raw audio, images, videos, and .osu text remain inside the original .osz archives under archives/. Metadata rows carry archive paths and member paths needed to locate those bytes.

Derived feature rows can be loaded the same way:

python
import pyarrow.dataset as ds

objects = ds.dataset(
    "./osu-everything/features/v1/tables/object_tokens_osu",
    format="parquet",
).to_table()

Updating

Clone or update source with Git:

bash
git clone https://huggingface.co/datasets/lekdan/osu-everything-tools ~/osu_model
cd ~/osu_model
git pull --ff-only

Build the tools:

bash
uv sync
cargo build --release -p osu_indexer -p osu_fetcher

Configure local osu! credentials in the dataset workspace:

bash
mkdir -p ~/osu_model_dataset
cp osu_fetcher.toml.example ~/osu_model_dataset/osu_fetcher.toml
# fill in ~/osu_model_dataset/osu_fetcher.toml

Run a normal update:

bash
UPLOAD=1 DOWNLOAD_CONCURRENCY=128 bash scripts/update_maps_v1.sh

The wrapper hydrates only bucket data/, schemas/, and state/fetcher/, fetches new or changed .osz files, ingests them, compacts metadata by default, regenerates latest views, and publishes the changed dataset prefixes. It does not download the historical archive corpus.

See docs/PIPELINE_RUNBOOK.md for operational details.

Derived ML features are built separately after compact metadata has been updated. See osu_feature_pipeline/README.md.

Source Changes

Make code and documentation changes in ~/osu_model, then use normal Git:

bash
git status --short
git add README.md docs scripts python crates schemas osu_feature_pipeline Cargo.toml Cargo.lock pyproject.toml uv.lock
git commit -m "describe the source change"
git push

Do not put runtime dataset state in the source commit. The source repo ignores archives/, data/, .fetcher/, .scratch/, incoming_osz/, logs/, target/, .venv/, and local credential files.

Licensing

Beatmaps frequently include copyrighted songs, backgrounds, videos, and storyboard assets. Do not publish this dataset publicly without the appropriate rights. Refer to the osu! beatmap licence before sharing.