Spatial9/GravityLLM

1---2language:3- en4license: apache-2.05library_name: transformers6pipeline_tag: text-generation7base_model: Qwen/Qwen2.5-1.5B-Instruct8tags:9- gravityllm10- spatial-audio11- immersive-audio12- spatial913- iamf14- instruction-tuning15- json16- lora17- qlora18- peft19- transformers20widget:21- text: |-22    INPUT:23    {24      "target_format": "iamf",25      "max_objects": 10,26      "style": "club",27      "section": "drop",28      "global": {"bpm": 128, "energy": 0.92},29      "stems": [30        {"id": "v1", "class": "lead_vocal", "lufs": -16.8, "transient": 0.25, "band_energy": {"low": 0.1, "mid": 0.6, "high": 0.3}, "leadness": 0.95},31        {"id": "k1", "class": "kick", "lufs": -10.5, "transient": 0.95, "band_energy": {"low": 0.8, "mid": 0.15, "high": 0.05}, "leadness": 0.25}32      ],33      "rules": [34        {"type": "anchor", "track_class": "lead_vocal", "az_deg": 0, "el_deg": 10, "dist_m": 1.6},35        {"type": "mono_low_end", "hz_below": 120}36      ]37    }38---39 40![GravityLLM banner](assets/heads9.png)41 42# GravityLLM43 44GravityLLM is a compact instruction-tuned model for **constraint-conditioned spatial scene generation**.  45It turns **music constraints + stem descriptors** into strict **Spatial9Scene JSON** for immersive audio pipelines such as IAMF, binaural, and bed-plus-object rendering workflows.46 47> **Status**48> This repository is **training-ready and Hub-ready**.  49> This includes code, schema, sample data, evaluation, and upload helpers.  50> It does **not** include fine-tuned weights yet. After training, upload the contents of your `outputs/...` folder as the actual model repo.51 52Demo at **[https://spatial9.ai/demo](https://spatial9.ai/demo)**53 54## What you will find in this repo55 56- Proper instruction fine-tuning with **prompt masking**, so the loss is applied to the target JSON instead of the instruction prefix.57- **LoRA** and **QLoRA** training paths for efficient fine-tuning on small-to-medium GPUs.58- Strict **JSON Schema** validation for production-safe outputs.59- Built-in **evaluation** for parse rate, schema-valid rate, object-budget pass rate, and anchor-rule pass rate.60- Clean **Hugging Face upload** helper with `upload_folder`.61- Ready-made **sample data**, **sample scene**, and **recommended training config**.62 63## Model contract64 65### Input66A structured payload describing:67 68- target format69- object budget70- style and section71- per-stem descriptors72- hard rules such as anchors, low-end centering, width targets, and masking constraints73 74### Output75A single valid JSON object matching `schemas/scene.schema.json`.76 77### Example input78```json79{80  "target_format": "iamf",81  "max_objects": 10,82  "style": "club",83  "section": "drop",84  "global": {"bpm": 128, "energy": 0.92},85  "stems": [86    {"id": "v1", "class": "lead_vocal", "lufs": -16.8, "transient": 0.25, "band_energy": {"low": 0.1, "mid": 0.6, "high": 0.3}, "leadness": 0.95},87    {"id": "k1", "class": "kick", "lufs": -10.5, "transient": 0.95, "band_energy": {"low": 0.8, "mid": 0.15, "high": 0.05}, "leadness": 0.25}88  ],89  "rules": [90    {"type": "anchor", "track_class": "lead_vocal", "az_deg": 0, "el_deg": 10, "dist_m": 1.6},91    {"type": "mono_low_end", "hz_below": 120}92  ]93}94```95 96### Example output97```json98{99  "version": "1.0",100  "bed": {"layout": "iamf", "loudness_target_lufs": -14.0, "room_preset": "club_medium"},101  "objects": [102    {103      "id": "v1",104      "class": "lead_vocal",105      "az_deg": 0,106      "el_deg": 10,107      "dist_m": 1.6,108      "width": 0.15,109      "gain_db": 0.0,110      "reverb_send": 0.18,111      "early_reflections": 0.22,112      "motion": [113        {"t": 0.0, "az_deg": 0, "el_deg": 10, "dist_m": 1.6},114        {"t": 1.0, "az_deg": 0, "el_deg": 10, "dist_m": 1.6}115      ]116    }117  ],118  "constraints_applied": [119    "anchor:lead_vocal@0/10/1.6",120    "mono_low_end<120Hz"121  ]122}123```124 125## Repository layout126 127```text128GravityLLM-HuggingFace-Repo/129├── README.md130├── LICENSE131├── Makefile132├── pyproject.toml133├── requirements.txt134├── train.py135├── infer.py136├── evaluate.py137├── upload_to_hub.py138├── assets/139│   └── gravityllm_banner.svg140├── configs/141│   └── recommended_train_args.json142├── data/143│   ├── train.jsonl144│   └── valid.jsonl145├── examples/146│   ├── sample_input.json147│   └── sample_output.json148├── schemas/149│   └── scene.schema.json150├── scripts/151│   ├── push_to_hub.sh152│   └── train_qlora.sh153└── tools/154    ├── make_synthetic_dataset.py155    └── validate_scene.py156```157 158## Quick start159 160### 1) Install161```bash162python -m pip install -r requirements.txt163```164 165### 2) Train with QLoRA166```bash167bash scripts/train_qlora.sh168```169 170Or run directly:171 172```bash173python train.py   --model Qwen/Qwen2.5-1.5B-Instruct   --train_file data/train.jsonl   --valid_file data/valid.jsonl   --output_dir outputs/GravityLLM-Qwen2.5-1.5B-S9   --max_length 2048   --num_train_epochs 3   --learning_rate 2e-4   --train_batch_size 1   --eval_batch_size 1   --gradient_accumulation_steps 16   --warmup_ratio 0.03   --save_steps 100   --eval_steps 100   --qlora --bf16174```175 176### 3) Generate a scene177```bash178python infer.py   --model_dir outputs/GravityLLM-Qwen2.5-1.5B-S9   --input_json examples/sample_input.json   --validate   --output_json outputs/sample_prediction.json179```180 181### 4) Evaluate182```bash183python evaluate.py   --model_dir outputs/GravityLLM-Qwen2.5-1.5B-S9   --data_file data/valid.jsonl   --report_path reports/eval_report.json184```185 186### 5) Validate any output187```bash188python tools/validate_scene.py schemas/scene.schema.json outputs/sample_prediction.json189```190 191## Push to the Hugging Face Hub192 193### From a trained output folder194```bash195python upload_to_hub.py   --folder_path outputs/GravityLLM-Qwen2.5-1.5B-S9   --repo_id YOUR_NAMESPACE/GravityLLM-Qwen2.5-1.5B-S9196```197 198### Or with the helper script199```bash200bash scripts/push_to_hub.sh outputs/GravityLLM-Qwen2.5-1.5B-S9 YOUR_NAMESPACE/GravityLLM-Qwen2.5-1.5B-S9201```202 203## Dataset format204 205Training files are JSONL with two fields per row:206 207```json208{209  "prompt": "GravityLLM: Output ONLY valid JSON matching the Spatial9Scene schema.\n\nINPUT:\n{...}",210  "completion": "{... valid Spatial9Scene JSON ...}"211}212```213 214The provided sample dataset is intentionally small. Replace it with your real production examples as soon as possible.215 216## Recommended data strategy217 218For a strong first release:219 2201. Collect a few hundred high-quality gold examples from expert-authored scenes.2212. Keep the schema stable and quantized.2223. Encode hard rules explicitly instead of relying on vague prose.2234. Run evaluation after every fine-tune.2245. Add a post-processor to enforce hard constraints if the runtime must be deterministic.225 226## Suggested training roadmap227 228### v0229- Small curated dataset230- QLoRA adapter231- Schema-valid JSON only232- Anchor and budget constraints233 234### v1235- More genres and sections236- Better masking and width rules237- Object motion patterns238- Automatic validation and repair loop239 240### v2241- Preference tuning on human A/B judgments242- A dedicated reward signal for clarity, masking avoidance, and translation safety243 244## Intended use245 246GravityLLM is designed for:247 248- music-tech pipelines249- Spatial9 scene authoring250- assisted immersive-audio layout generation251- IAMF-ready authoring workflows252- renderer-side JSON generation253 254## Limitations255 256- This repo does not include trained weights out of the box.257- The model only knows what you teach it through your dataset.258- Raw audio is not consumed directly here; the training pipeline expects structured stem features.259- Production systems should still validate outputs and optionally apply a rule-based correction pass.260 261## Safety and reliability262 263- Always validate generated scenes against the JSON schema.264- Keep low-end centering as a hard rule outside the model if that is non-negotiable.265- Treat the model as a scene proposal engine, not an oracle.266 267## License268 269This repository is released under Apache-2.0.270