Aluode/SplatWorld

1---2title: SplatWorld3emoji: 🌊4colorFrom: indigo5colorTo: pink6sdk: gradio7sdk_version: 5.49.18app_file: app.py9pinned: false10license: cc-by-nc-4.011tags:12  - gabor13  - wave-optics14  - vae15  - generative16  - faces17  - celeba18  - onnx19---20 21<!-- ^^^ Hugging Face Space header. On GitHub, delete this block (or keep it —22     GitHub renders it as an empty comment). Adjust sdk_version to a version23     Hugging Face currently offers if the build complains. -->24 25# SplatWorld26 27**202,599 CelebA faces compressed into one 7.2 MB decoder — and it doesn't store a single pixel.**28 29SplatWorld is a small VAE whose decoder doesn't paint images. It maps a 128‑dimensional point `z` to **256 Gabor wave packets** and lets their interference *become* the picture. A face is a phase‑locked standing wave. The "fire" you see drifting between faces is those same waves coming unlocked. This repo gives you two ways to walk around inside that world.30 31---32 33## Two apps34 35| | run it | what it is |36|---|---|---|37| **`app.py`** (Gradio) | `python app.py` → opens in a browser | Runs anywhere, including **Hugging Face Spaces**. Surf with sliders, scrub the zoom, render a zoom video, sample a gallery. This is the Space entry point. |38| **`explore_desktop.py`** (OpenCV) | `python explore_desktop.py` | The richer **local** tool: a live window with real mouse‑drag surfing and a hands‑free automated zoom. Needs a display, so it can't run on Spaces. |39| **`splat_atlas.py`** | `python splat_atlas.py --dump` then `--browse` | Bakes thousands of thumbnails to disk to survey the whole latent space; click any tile back to a full‑res face. |40| **`splat_trainer2.py`** | `python splat_trainer2.py --data_dir …` | Rebuild the model from your own faces (needs PyTorch + a CUDA GPU). See *Train your own model* below. |41 42Both need `splat_decoder.onnx` (7.2 MB) sitting next to them. The Gradio app still boots without it (showing mock noise), so a Space never hard‑fails on startup — but add the model to see faces.43 44---45 46## Quick start (browser app)47 48```bash49pip install -r requirements.txt50# put splat_decoder.onnx next to app.py, then:51python app.py                 # http://127.0.0.1:786052```53 54On **Hugging Face**: create a Gradio Space, upload these files plus `splat_decoder.onnx`, and it runs. The header block at the top of this README is the Space config.55 56## Quick start (desktop app)57 58```bash59pip install -r requirements-desktop.txt60python explore_desktop.py     # live OpenCV window61python explore_desktop.py --selftest62python explore_desktop.py --gallery 64   # headless: dump faces to ./gallery/63```64 65Desktop keys: `1` zoom · `2` surf · `3` atlas · `H` theory · `B` gallery · `R` record · `S` save · `Q` quit. In **surf**, drag to morph, wheel to dive/rise, `SPACE` for a new face, `N` to re‑roll the drag plane.66 67---68 69## The atlas (`splat_atlas.py`)70 71The atlas is a heavier, separate tool. It systematically **bakes thousands of thumbnails to disk** (with every `z` saved) so you can survey the whole space at a glance and click any tile back to a full‑res face.72 73```bash74python splat_atlas.py --dump --gb 2   # build it ONCE (writes ./atlas/, a few GB)75python splat_atlas.py --browse        # click a tile -> full res + z;  n/p flip pages76python splat_atlas.py --analyze       # fine-band energy vs |z|: the "departure curve"77```78 79`--browse` will say *"no atlas found — run --dump first"* until you've dumped. Once `./atlas/` exists, `app.py`'s zoom automatically uses the real core faces in it as waypoints.80 81---82 83## How it works84 85The decoder is a tiny MLP: `z (128-D) → 256 packets × 11 numbers`. Each packet is a Gabor atom — **position, size (σ), orientation (θ), frequency, and a complex `(a, b)` amplitude per color channel**, i.e. the cosine weight and the sine weight of a little oriented wave. The renderer drops all 256 vibrating drumheads on a 96×96 canvas and sums their ripples with explicit `env·cos` and `env·sin` math. No convolutions, no pixels — just interference.86 87- **Core** (`|z| < 15`): the atoms **phase‑lock**. Peaks and troughs align to cancel everywhere except along a sharp eyebrow or a soft cheekbone. A standing wave — a face.88- **Fire** (`|z| > 35`): no training data lives out here, so the decoder stops orchestrating. Envelopes wander, phases drift, packets beat against each other. That decorrelating soup is what you're flying through between faces.89 90Training is coarse‑to‑fine and you can watch it happen: the **envelope** code (where the mass sits) commits first, the **carrier** code (orientation and phase) commits second. Early in training the eyebrows are flat horizontal blobs (envelope only); once the coarse loss plateaus the model finally commits an orientation and the brows snap into their characteristic slant. Amplitude first, phase later.91 92---93 94## The space between faces (the theory)95 96Why is there *anything* between two faces? Because moving a feature from A to B is **transport**, and transport is where linear models break.97 98In a fixed additive basis, the only way to move an eyebrow is to fade one atom out while fading another in. Mid‑way both exist, their phases fight, and you get interference. **The fire is that fight made visible.** This is not new — it's 1990s technology. *Eigenfaces* built face‑space as linear combinations of basis faces in 1991, and interpolating between two of them produced exactly these ghostly double‑exposures. A purely additive "image model by linear transfer" is ancient, and the fire is its failure mode frozen in place.99 100There's a loophole worth naming, because it's the direction this whole line of work keeps pointing at. A complex `(a, b)` atom can **translate a feature by rotating its phase** — the Fourier shift theorem — instead of crossfading amplitudes. Phase‑transport moves the wave without ever destroying it, so it leaves no ghost. The atoms here already carry that complex structure; a "good" version of this world would move features by rotating phases, not sliding coefficients.101 102And the size of the gap is a tug‑of‑war you can feel in the trainer. Squeeze faces together (raise the VAE's β) and the morphs get buttery but identities collapse toward a bland mean. Push them apart (low β) and every face is crisp but the fire between them is vast. Crisp‑and‑dense at once — density without collapse — is the actual open problem.103 104> If you want the quantitative version: `splat_atlas.py --analyze` measures fine‑band energy as a function of `|z|`. That curve is "the splats appear and grow," plotted. The local geometry of the gaps (how fast the image changes per step of `z`, direction by direction) is a Riemannian pullback metric — mapping it would tell you exactly where the fire is thick and where faces slide smoothly into each other.105 106---107 108## Troubleshooting109 110**`cv2.error ... buf.u == m.u` on OpenCV 5.** OpenCV 5's new dnn engine crashes111on this graph when the browser app runs the model in a worker thread. Two fixes,112either works:113 114- **Install onnxruntime** (recommended, and what the Space uses):115  `pip install onnxruntime`. The app prefers it automatically and it sidesteps116  OpenCV entirely.117- If you only have OpenCV, the app now builds a **per-thread** net and prefers118  the **classic** engine, which avoids the assertion. Just update to this119  version of `app.py`.120 121The status line under the title tells you which backend loaded122(`onnxruntime`, `opencv-dnn`, or `mock`).123 124**The Space shows mock noise.** `splat_decoder.onnx` isn't next to `app.py`. Add125it (commit it to the Space repo, or use Git LFS).126 127---128 129## Train your own model (`splat_trainer2.py`)130 131The trainer is included so you can rebuild `splat_decoder.onnx` from your own132folder of faces (or retrain on CelebA). It needs **PyTorch and a CUDA GPU** for133real speed; the model here was trained on a 12 GB card.134 135```bash136pip install -r requirements-train.txt      # torch build must match your CUDA137 138# 1) train  (first run caches the dataset once, then trains from the GPU-resident tensor)139python splat_trainer2.py --data_dir /path/to/faces --beta 0.0005140 141# 2) export the trained checkpoint to the ONNX the explorers use142python splat_trainer2.py --export          # -> splat_decoder.onnx143 144# 3) sanity-check the whole pipeline on CPU, no data or GPU needed145python splat_trainer2.py --smoke146```147 148How it's fast: it **caches** the folder to a `uint8` array once (decoding 200k149JPEGs every epoch was the real bottleneck), keeps the whole **dataset resident on150the GPU** (≈5.6 GB at 96px, no DataLoader, no per-step host→device copy), and151trains in **gradient steps** with a vectorized renderer. `--smoke` was verified152end-to-end (loop-vs-vectorized renderer parity, a short train, export, and153`cv.dnn`↔torch agreement to 1e-4 across a batched dynamic axis).154 155Useful flags (defaults in brackets):156 157| flag | meaning |158|---|---|159| `--data_dir` [`./faces`] | folder of images (jpg/png/bmp/webp), center-cropped and resized |160| `--image_size` [`96`] | render resolution — bigger costs VRAM fast |161| `--num_packets` [`256`] | Gabor atoms per image |162| `--steps` [`30000`] · `--batch` [`96`] · `--lr` [`3e-4`] | training length / batch / learning rate |163| `--beta` [`1.0`] · `--beta_warmup_steps` [`3000`] | KL weight and its ramp. **Low β (e.g. `0.0005`) is what gives varied faces**; high β collapses to a mean face |164| `--gamma_floater` [`0.02`] · `--sigma_ref` [`0.03`] | anti-"floater" penalty: taxes envelopes thinner than `sigma_ref` (`0` disables) |165| `--checkpointing` | halve VRAM, double renderer compute — only if you OOM |166| `--out` [`./runs/splat2`] | where `model2.pt` and the recon/sample grids are written |167 168The export always writes the input/output names (`z_latent` / `rendered_image`,169opset 17, dynamic batch) that every tool in this repo expects, so a freshly170trained model is a drop-in replacement.171 172> **Note on ONNX + OpenCV 5.** The export uses a dynamic batch axis. OpenCV 5's173> dnn importer can't parse it, so the explorers load the model with174> **onnxruntime** (see Troubleshooting). The trainer's own `--smoke` uses175> `cv.dnn` on a tiny graph and passes; the full 256-packet model needs176> onnxruntime at inference time.177 178---179 180## Model card / provenance181 182| | |183|---|---|184| **Data** | CelebA, 202,599 aligned face images |185| **Resolution** | 96×96 (chosen to fit in VRAM, not for quality) |186| **Latent** | 128‑D |187| **Packets** | 256 Gabor atoms, 11 params each |188| **Params** | ~5.74M |189| **Training** | 30,000 steps, batch 96, cosine LR, β warm‑up |190| **Export** | ONNX, 7.2 MB, dynamic batch axis (many `z` per forward) |191| **Inference** | CPU‑friendly via OpenCV `cv2.dnn`; no GPU or PyTorch required |192 193The trainer itself is not in this repo — this repo is the *explorer*. (Training code lives with the model line, e.g. TheSplat5.)194 195---196 197## Honest ledger198 199*Do not hype. Do not lie. Just show.*200 201**What's real and shown here**202- The decoder genuinely reconstructs and samples CelebA‑like faces from a 7.2 MB file, live.203- The core→fire shell structure, and the coarse‑to‑fine (envelope‑then‑carrier) commitment, are directly observable in the tools.204- Zoom's `z`‑path is C0 across identity hand‑offs and across the scale wrap; the surf tangent axes are provably orthonormal (see `--selftest`).205 206**What's honest to admit**207- 96×96 is a memory limit, not an aesthetic. Hair and fine detail struggle; sampled faces skew toward the population mean.208- The "phase‑locked standing wave" framing is a faithful description of a Gabor renderer, not a claim of new physics. The physics/AI resonance here is a *representation* echo (complex amplitudes = phasors), not a discovery about nature.209- SURF explores a 2‑plane of the tangent space at a time. It does not give you free access to all 128 dimensions at once; press `N` for a new plane.210- The phase‑transport loophole above is a *direction*, not something this particular model already does well.211 212**License / data terms.** CelebA is released for **non‑commercial research use** — check the dataset's own terms before you do anything with the outputs. This repo is offered under CC BY‑NC 4.0 to stay consistent with that.213 214---215 216## Credits217 218Built by [Antti Luode](https://github.com/anttiluode) with several AI systems as thinking partners over a couple of long sessions. The interference/eigenface/phase‑transport framing came out of those conversations; the code and the honest ledger are meant to let you check all of it yourself.219