Reverb/open3dforge

1# Open3DForge — Full Build Plan2 3> **Project:** Personal image-to-game-ready 3D asset pipeline4> **Owner:** Basel · Solo dev for "What Remains" (UE5.7)5> **Hosting:** Single HF Pro Space, ZeroGPU H200, 25 min/day quota6> **SDK:** Gradio 6.x (currently 6.14.0)7> **Repo:** `Reverb/open3dforge`8 9---10 11## Architectural Decisions (Locked In)12 13These were debated and decided during early development. Don't relitigate them mid-build.14 151. **One Space, not multiple.** Option B: vendor TRELLIS.2 + Hunyuan3D-2 + UniRig into this Space rather than orchestrating across multiple Spaces via `gradio_client`. Larger repo, but single deployment, no inter-Space latency, no auth juggling.16 172. **HF Space only, no local fallback.** Don't build a path for running on the RTX 4070. Quota is enough for personal use.18 193. **Gradio 6.x.** Match the `sdk_version` in `README.md`. No upper bound in `requirements.txt`.20 214. **UE5-first export defaults.** DirectX normals, ORM packing, cm units, Z-up, `SM_/SK_/T_` naming.22 235. **Drop the custom website.** Standard Gradio tabs. `gradio.Server` is not worth the work for one user.24 256. **Drop CHORD.** Research-only license. Use TRELLIS.2's own metallic/roughness volume attributes instead, which are already correct and license-compatible.26 277. **nvdiffrast for all baking.** Not Blender headless. Fast (~2-5s per bake), GPU-based, fits inside `@spaces.GPU`, no apt install needed beyond what TRELLIS already requires.28 298. **Solo-user workspace pattern.** One `workspace/current/` folder. No session IDs, no multi-tenancy.30 31---32 33## Reference Code: Pipeline Stage Order34 35```36INPUT: 1-4 images37   │38   ├── rembg (background removal, CPU)39   │40   ▼41[Stage 1] GENERATION                          [GPU]42   ├── TRELLIS.2 (hard surface) or Hunyuan3D-2 (organic)43   ├── SAVE high_poly.glb (kept for normal baking)44   └── Extract: albedo + metallic + roughness volume attrs45   │46   ▼47[Stage 2] POST-PROCESSING                     [CPU + GPU baking]48   2A  Mesh repair             pymeshfix          CPU49   2B  Geometry cleanup        PyMeshLab          CPU50   2C  Decimation                                 CPU51        ├── Preview: fast-simplification52        └── Final:   PyMeshLab quality pass53   2D  Symmetry (characters)   PyMeshLab          CPU54   2E  UV unwrap               xatlas             CPU55        (texels_per_unit packing)56   2F  Normal bake             nvdiffrast         GPU57        DX + GL outputs58   2G  Albedo bake             nvdiffrast         GPU59        vertex color → UV atlas60   2H  Material bake           nvdiffrast         GPU61        TRELLIS volume attrs → UV62   2I  AO bake                 nvdiffrast         GPU63        ray-occlusion → UV64   2J  Texture inpaint (opt)   SDXL inpaint       GPU65        hidden UV regions66   2K  Channel pack            numpy              CPU67        Unreal ORM / Unity MetSmooth68   2L  LOD generation          PyMeshLab          CPU69        LOD0/LOD1/LOD2 only (UE5 HLOD handles billboards)70   2M  Collision mesh          CoACD              CPU71   2N  Pivot correction        trimesh            CPU72   2O  Scale validation        trimesh            CPU73   │74   ▼75[Stage 3] AUTO-RIGGING (Optional)             [GPU]76   UniRig → rigged.glb / rigged.fbx77   │78   ▼79[Stage 4] EXPORT80   UE5 preset (default) → DX normals + ORM81   Naming: SM_/SK_/T_ convention82   → export_AssetName_UE5.zip83```84 85---86 87## Milestone Plan88 89Each milestone is sized to be one focused work session. Push at the end of each, verify, then move on.90 91---92 93### ✅ Milestone 1 — Foundation (COMPLETE)94 95**Status:** Deployed and verified. ZeroGPU smoke test passes.96 97What got built:98- HF Space scaffolded with Gradio 6 + ZeroGPU99- 5 tabs: Generate / Post-Process / Auto-Rig / Export / Presets + Diagnostics100- `workspace/` folder pattern (current/exports/presets/history)101- `src/workspace.py` — AssetState, preset save/load102- `src/quota.py` — daily quota tracking103- `src/ui_helpers.py` — status bar, asset summary, viewer model picker104- `gr.Model3D` viewer wired up105- Pipeline stubs returning placeholder messages106- Diagnostics tab with `@spaces.GPU` smoke test107 108Files in repo:109```110open3dforge/111├── README.md             ← sdk_version: 6.14.0112├── requirements.txt      ← gradio>=5.0, spaces, numpy, pillow113├── .gitignore114├── app.py                ← main entry, all UI wiring115├── src/116│   ├── __init__.py117│   ├── workspace.py118│   ├── quota.py119│   └── ui_helpers.py120└── workspace/121    ├── current/.gitkeep122    ├── exports/.gitkeep123    ├── presets/.gitkeep124    └── history/.gitkeep125```126 127---128 129### 🟡 Milestone 2 — Stage 1: TRELLIS.2 Generation (NEXT)130 131**Goal:** Real image-to-3D generation working end-to-end. Upload image → get a GLB in the viewer.132 133**Approach:** Option B — duplicate the microsoft/TRELLIS.2 Space, merge its contents into our repo, then refactor app.py to integrate with our tab structure.134 135**Step-by-step:**136 1371. **Duplicate microsoft/TRELLIS.2 Space** to get a known-good baseline:138   - On HF: `huggingface.co/spaces/microsoft/TRELLIS.2` → Duplicate this Space → name it `open3dforge-trellis-staging`139   - This is a *staging* copy — we don't deploy it, we just clone it locally for the merge140   - Confirm it builds and runs in your duplicate before touching anything141 1422. **Clone both repos locally:**143   ```bash144   git clone https://huggingface.co/spaces/Reverb/open3dforge145   git clone https://huggingface.co/spaces/baselanaya/open3dforge-trellis-staging146   ```147 1483. **Copy TRELLIS.2 assets into open3dforge:**149   ```bash150   cp -r open3dforge-trellis-staging/trellis2/ open3dforge/151   cp -r open3dforge-trellis-staging/assets/ open3dforge/152   cp open3dforge-trellis-staging/autotune_cache.json open3dforge/153   cp open3dforge-trellis-staging/packages.txt open3dforge/154   ```155   This gives us the vendored `trellis2/` Python package, HDRI envmaps, FlexGemm cache, and apt deps.156 1574. **Merge requirements.txt:**158   Combine the TRELLIS.2 requirements with our existing ones. Add to `requirements.txt`:159   ```160   # TRELLIS.2 deps (from microsoft/TRELLIS.2 Space)161   torch162   torchvision163   cv2 / opencv-python-headless164   imageio165   imageio-ffmpeg166   rembg167   # plus the custom wheels they install at build time168   ```169   Copy theirs verbatim and add to ours. Inspect the resolved `requirements.txt` in the staging duplicate first.170 1715. **Refactor app.py to integrate the TRELLIS handlers:**172   - Move TRELLIS pipeline init to module level (per ZeroGPU rules — must be on CUDA at module-level)173   - Wrap their `image_to_3d` + `extract_glb` functions as the implementation of our existing `stub_generate` handler174   - Update the Generate tab to match TRELLIS parameter names (resolution, ss_sampling_steps, etc.)175   - Hide most TRELLIS knobs behind the "Advanced" accordion; expose only Quality preset + Seed at top level176   - Keep our quality presets (Fast/Balanced/Hero) mapping to their parameter sets177   - Hook the output GLB into `workspace.get_state().raw_gen_glb` and save the high-poly separately178 1796. **Critical: save `high_poly.glb` before decimation.** TRELLIS's `extract_glb` calls `o_voxel.postprocess.to_glb(decimation_target=...)`. We need to call it once with no decimation (or a very high target like 16M faces — the nvdiffrast limit they use) to get the high-poly we'll bake from in Stage 2, then call it again with the user's chosen decimation_target for the working low-poly.180 1817. **Update workspace state on success:**182   ```python183   state = workspace.get_state()184   state.high_poly_glb = Path("workspace/current/high_poly.glb")185   state.raw_gen_glb = Path("workspace/current/raw_gen.glb")186   state.face_count = len(mesh.faces)187   state.vertex_count = len(mesh.vertices)188   state.model_used = "TRELLIS.2"189   ```190 1918. **Test:**192   - Push to Space193   - Wait for build (~10-15 min due to CUDA wheels compiling)194   - Upload a test image195   - Confirm the GLB appears in the viewer196   - Confirm Diagnostics quota tracker shows time consumed197 198**Quality presets to wire up (map to TRELLIS params):**199 200| Preset | resolution | ss_steps | shape_steps | tex_steps | Expected time |201|---|---|---|---|---|---|202| Fast | 512 | 8 | 8 | 8 | ~30s |203| Balanced | 1024 | 12 | 12 | 12 | ~60s |204| Hero | 1536 | 16 | 16 | 16 | ~90s |205 206**Risk mitigation:**207- TRELLIS.2 build can fail in many ways (CUDA wheel compilation, flash-attn install). If a build fails, check the build logs for which wheel failed. The staging duplicate is the reference — if it built there, the issue is in *your* merge.208- Don't move anything into `@spaces.GPU` functions that should be at module level. Pipeline init goes at module level.209 210---211 212### Milestone 2b — Hunyuan3D-2 Alternative Generator213 214**Goal:** Second generator option for organic shapes (characters, creatures).215 216**Approach:** Same duplicate-and-vendor pattern as Milestone 2.217 2181. Duplicate `tencent/Hunyuan3D-2` to staging Space2192. Clone, copy the `hy3dgen/` package into open3dforge2203. Merge requirements (most overlap with TRELLIS.2 — torch, diffusers)2214. Add Hunyuan pipeline init at module level2225. The model dropdown in the Generate tab routes between `image_to_3d_trellis()` and `image_to_3d_hunyuan()`2236. Hunyuan needs 16GB VRAM — fits alongside TRELLIS in H200's 70GB but only load one at a time via lazy module-level guards224 225**Decision deferred to this milestone:** Whether to keep both models in VRAM at module load (faster, more memory) or lazy-load per call (slower first call, less memory). Test both.226 227---228 229### Milestone 3 — Stage 2A-2C: Mesh Cleanup230 231**Goal:** Working CPU-side mesh repair, cleanup, and decimation with live preview.232 233**Dependencies to add:**234```235trimesh[easy]236pymeshfix237pymeshlab238fast-simplification239```240 241**Files to create:**242- `src/stages/__init__.py`243- `src/stages/stage2_repair.py` — pymeshfix wrapper244- `src/stages/stage2_cleanup.py` — PyMeshLab filter chain245- `src/stages/stage2_decimate.py` — both fast-simplification (preview) and PyMeshLab (final)246 247**UI work in app.py:**248- Wire the existing checkboxes/sliders in Tab 2 to call the real implementations249- Live preview: slider `.change()` event fires `fast-simplification`, updates face count display250- Run button: actually runs full pipeline on the current GLB251 252**Workspace state updates:**253- `state.repaired_glb`, `state.cleaned_glb`, `state.low_poly_glb` all get populated as steps complete254 255**Test criteria:**256- Generate a TRELLIS asset (50k faces)257- Run repair → no errors258- Run cleanup → no errors259- Set decimation slider to 10k → live preview updates face count260- Click "Run final" → produces low_poly.glb at 10k faces261- Viewer auto-refreshes to show the cleaned mesh262 263---264 265### Milestone 4 — Stage 2D-2E: Symmetry + UV Unwrap266 267**Goal:** Symmetry enforcement + xatlas UV unwrapping with consistent texel density.268 269**Dependencies to add:**270```271xatlas272```273 274**Files to create:**275- `src/stages/stage2_symmetry.py` — PyMeshLab `apply_filter_mesh_symmetrize`276- `src/stages/stage2_uv.py` — xatlas with `texels_per_unit` packing277 278**UI work:**279- Symmetry: off / bilateral-X / bilateral-Y / radial dropdown280- UV: atlas resolution, texels_per_unit, padding281 282**Test criteria:**283- Run on a human-character GLB → symmetry produces clean mirror284- UV unwrap produces `unwrapped.glb` with valid UV0 coords visible if you inspect via trimesh285- No overlapping UV islands (check with PyMeshLab's quality measure)286 287---288 289### Milestone 5 — Stage 2F: Normal Baking with nvdiffrast290 291**Goal:** High-poly → low-poly normal map baking, GPU-accelerated, 2-5 second bakes.292 293**Dependencies to add:**294- `nvdiffrast` (already installed via TRELLIS.2 wheels — verify in the staging duplicate)295 296**Files to create:**297- `src/stages/stage2_bake_normal.py` — full nvdiffrast pipeline298 299**Algorithm (from the plan doc):**300```python301@spaces.GPU(duration=60)302def bake_normal_map(high_poly_path, low_poly_path, uv_coords, map_size=2048):303    ctx = dr.RasterizeCudaContext()304    # 1. UV → clip space305    # 2. Rasterize low-poly UVs → per-pixel world position + tri ID306    # 3. For each pixel: nearest-on-surface from high-poly307    # 4. Sample high-poly normal at that point308    # 5. Transform to tangent space (low-poly tangent frame)309    # 6. Pack RGB [0,1], save PNG310    # 7. Dilate edges past UV island boundaries311```312 313**Output:** Two PNGs — `normal_gl.png` and `normal_dx.png` (DX has Y-flipped green channel).314 315**Test criteria:**316- Run on TRELLIS character output (50k high-poly → 10k low-poly)317- Bake completes in <10 seconds318- Open the normal map in any image viewer — should be bluish/purple with surface detail visible319- Both DX and GL versions are produced320- Quota shows 5-10 seconds consumed321 322---323 324### Milestone 6 — Stage 2G-2I: Albedo, Material, AO Baking325 326**Goal:** Three more nvdiffrast bakes producing the full PBR texture set.327 328**Files to create:**329- `src/stages/stage2_bake_albedo.py`330- `src/stages/stage2_bake_material.py` — uses TRELLIS.2's stored metallic+roughness attrs331- `src/stages/stage2_bake_ao.py` — ray-occlusion in hemisphere332 333**Key reuse:** Same nvdiffrast rasterization pattern as Milestone 5 — refactor that code into a shared helper `_rasterize_uv_atlas()` in `src/stages/_baking_helpers.py`.334 335**Workspace state:** All texture paths populated on the AssetState.336 337**Test criteria:**338- All four maps (normal, albedo, metallic, roughness, AO) viewable as PNG thumbnails in Tab 2339- Total Stage 2 baking time < 30 seconds for a Balanced-quality asset340 341---342 343### Milestone 7 — Stage 2J: SDXL Inpainting for Hidden UVs344 345**Goal:** Detect stretched/synthetic UV regions and inpaint them with SDXL.346 347**Dependencies to add:**348```349diffusers350accelerate351safetensors352```353 354**Files to create:**355- `src/stages/stage2_inpaint.py`356  - `detect_hidden_regions(albedo, uvs, faces)` — variance analysis357  - `inpaint_hidden_uvs(...)` — SDXL inpainting pipeline358 359**UI:** Toggle off by default (costs ~30s quota). Prompt input. Strength slider.360 361**Test criteria:**362- Generate an asset with a clear "back side" (e.g., a humanoid character)363- Without inpainting: back of character has visible texture stretching364- With inpainting: back is plausibly filled in365- Quota cost: ~30s per inpaint366 367---368 369### Milestone 8 — Stage 2K-2O: Finalization Steps370 371**Goal:** Channel packing, LODs, collision, pivot, scale — all CPU-side, fast.372 373**Dependencies to add:**374```375coacd==1.0.4376```377 378**Files to create:**379- `src/stages/stage2_channel_pack.py` — numpy ORM / MetallicSmoothness packing380- `src/stages/stage2_lods.py` — PyMeshLab quality-aware LOD0/1/2381- `src/stages/stage2_collision.py` — CoACD with `trimesh.convex_hull` fallback382- `src/stages/stage2_pivot.py` — bottom_center / geometric_center / custom383- `src/stages/stage2_scale.py` — height presets, UE5 cm units384 385**UI:** All controls already scaffolded in Milestone 1's Post-Process tab. Just wire to real implementations.386 387**Test criteria:**388- ORM packed as RGB with AO/Roughness/Metallic in correct channels389- LOD0/LOD1/LOD2 all generated, all share same UV layout390- Collision mesh has <1% the triangle count of LOD0391- Pivot at bottom_center for a generated human character results in feet at world origin Y=0392- Scale: human asset is 1.8m tall = 180cm in UE5 export393 394---395 396### Milestone 9 — Stage 3: UniRig Auto-Rigging397 398**Goal:** Generate a skeleton + skinning weights for character meshes.399 400**Approach:** Same vendor-the-Space pattern as Milestone 2.401 4021. Duplicate `MohamedRashad/UniRig` Space → staging4032. Verify it builds in the staging duplicate4043. Copy `UniRig/` package into our repo4054. Merge requirements4065. Wire to the Auto-Rig tab handler4076. Output: rigged FBX (UE5 default) or GLB408 409**Test criteria:**410- Run on a humanoid character (after full Stage 2 processing)411- Output FBX imports into UE5 as a Skeletal Mesh412- Drag into Mixamo → animations auto-attach correctly413 414---415 416### Milestone 10 — Stage 4: UE5 Export417 418**Goal:** Bundle everything into a UE5-ready zip with proper naming and packing.419 420**Dependencies to add:**421```422pygltflib423```424 425**Files to create:**426- `src/stages/stage4_export.py`427  - `export_ue5(asset_state, asset_name, asset_type) → zip_path`428  - Handles FBX conversion via trimesh429  - Applies naming convention (`SM_`, `SK_`, `T_`)430  - Writes ORM-packed textures to correct paths431  - Zip + drop in `workspace/exports/`432 433**Engine presets (only UE5 fully implemented):**434- UE5: FBX, DX normals, ORM, Z-up, cm — the default435- Unity HDRP: FBX, GL normals, MetallicSmoothness, Y-up, m — stub for later436- Godot/Blender/Web: stubs437 438**Test criteria:**439- Export a character → unzip → 6-7 files following naming convention440- Import to UE5: drag-drop the zip's contents → no warnings, materials auto-create from textures441- Both Static Mesh and Skeletal Mesh paths work442 443---444 445### Milestone 11 — Presets System446 447**Goal:** Save and load named parameter configurations across tabs.448 449**Files to update:**450- `src/workspace.py` — already has `save_preset/load_preset/delete_preset`, just needs the JSON schema fleshed out451- `app.py` — wire the Presets tab's Save button to actually read all current tab values452 453**Schema:**454```json455{456  "name": "character_UE5_hero",457  "stage1": { ... },458  "stage2": { ... },459  "stage3": { ... },460  "stage4": { ... }461}462```463 464**Ship five default presets:**465- `character_UE5_hero.json`466- `character_UE5_npc.json`467- `prop_UE5_hero.json`468- `prop_UE5_standard.json`469- `environment_UE5_background.json`470 471---472 473### Milestone 12 — Polish & Production Hardening474 475- Error handling on every stage (don't crash the app, show clear error in UI)476- Progress bars during long ops (`gr.Progress(track_tqdm=True)`)477- Quota cost shown *before* each GPU operation (warning if it would exceed remaining)478- Game-ready checklist passes shown before allowing Export479- Asset history sidebar (last 5 generated assets with thumbnails)480- Session cleanup of `workspace/current/` on new generation481 482---483 484## Working with Claude Code485 486When you continue in Claude Code, you'll have the full repo locally. Key things to remember:487 488### Project conventions489 4901. **Each stage = its own module** in `src/stages/`. Don't dump pipeline logic into `app.py`.4912. **Workspace state is the single source of truth.** Every stage reads from and writes to `workspace.get_state()`.4923. **GPU functions live where they're needed**, not all in app.py. The `@spaces.GPU` decorator works in any file as long as `spaces` is imported.4934. **No `if __name__ == "__main__":` on `demo.launch()`.** HF Spaces imports app.py at module level.4945. **Gradio 6 specifics:**495   - `theme` and `css` go in `launch()`, not `Blocks()`496   - `show_api` is gone — use `footer_links=["gradio", "settings"]`497   - `api_visibility` replaces `api_name=False` on events4986. **The 3 global components** (`viewer`, `summary`, `status_bar`) get refreshed via `_global_refresh()` chained off every pipeline action button. Don't forget to add new buttons to that list.499 500### Useful commands501 502```bash503# Pull the latest Space state504cd open3dforge505git pull506 507# Make changes, syntax-check before push508python -c "import ast; ast.parse(open('app.py').read())"509 510# Push to deploy511git add -A512git commit -m "Milestone N: <stage>"513git push514 515# Watch build/runtime logs at:516# https://huggingface.co/spaces/Reverb/open3dforge?logs=container517```518 519### Common HF Space build failures (we've hit these)520 521| Symptom | Cause | Fix |522|---|---|---|523| `Cannot install gradio<X and gradio==Y` | `sdk_version` in README conflicts with requirements.txt pin | Remove version pin in requirements.txt or update README's sdk_version |524| `Blocks.launch() got an unexpected keyword argument 'X'` | Gradio 6 removed parameter | Check Gradio 6 migration guide for replacement |525| `When localhost is not accessible` | `demo.launch()` wrapped in `if __name__ == "__main__"` | Move to module level |526| CUDA wheel compile failures | Mismatched torch/CUDA versions | Match TRELLIS.2 staging duplicate's exact pins |527| OOM during model load | Multiple large models loaded at module level | Lazy-load with module-level guards inside `@spaces.GPU` |528 529### Useful resources530 531- **Gradio 6 migration guide:** https://www.gradio.app/main/guides/gradio-6-migration-guide532- **ZeroGPU docs:** https://huggingface.co/docs/hub/spaces-zerogpu533- **TRELLIS.2 reference Space:** https://huggingface.co/spaces/microsoft/TRELLIS.2534- **Hunyuan3D-2 reference Space:** https://huggingface.co/spaces/tencent/Hunyuan3D-2535- **UniRig reference Space:** https://huggingface.co/spaces/MohamedRashad/UniRig536 537---538 539## Constraints to Remember540 541- **Daily quota:** 1500s (25 min) of H200 time per day. Plan asset iteration accordingly.542- **VRAM budget:** ~70GB per workload. TRELLIS.2 alone is 24GB; UniRig is 8GB; SDXL inpaint is 8GB. Don't load all at once.543- **Function timeout:** Default `@spaces.GPU` duration is 60s. Override with `duration=N` for longer ops (Stage 1 generation, AO bake high quality).544- **Build time:** With TRELLIS.2 vendored + CUDA wheels, expect 10-15 min builds. Cache hits will be ~3 min.545- **Repo size:** Will grow large with vendored models + HDRIs. Git LFS may be needed for the autotune_cache.json (~1MB) and wheel files (~100MB+). HF Spaces handles this via Xet storage automatically.546 547---548 549*Plan version 3.0 — May 15, 2026*550*Last action completed: Milestone 1 deployed, ZeroGPU smoke test passing*551*Next action: Milestone 2 — duplicate microsoft/TRELLIS.2 staging Space, merge into open3dforge*