Ronaldo-GOAT/pose6daug

1# INSTALL — environments and third-party dependencies2 3This tree is a **code-only snapshot** of the simulator-based action-augmentation4pipeline (`/lp-dev/jonghoon/rebuttal/sim_action_aug`, symlinked as5`/lp-dev/jonghoon/sim_action_aug`). No model weights, no third-party repos and no6datasets are vendored here. Everything below is what the pipeline actually ran with7(versions read from the live environments on 2026-08-25; anything not directly8verifiable is marked *unverified*).9 10## 1. Python environments actually used11 12The pipeline deliberately reuses pre-existing shared conda envs plus four13project-local venvs (originals under14`/lp-dev/jonghoon/rebuttal/sim_action_aug/envs/`, 6.8 GB, NOT copied here).15 16| env (interpreter path) | python / torch | used for |17|---|---|---|18| `/data/nvidia/gripper_augmentator/conda-envs/sam3d/bin/python` | 3.11.0 / torch 2.5.1+cu121 | prep (frames, ZED depth via pyzed+TRT8 shim, teleop parse, calib bundle), recon (SAM3D-Objects, open3d, pyrender, nvdiffrast), native-twin build |19| `/data/nvidia/gripper_augmentator/conda-envs/sam2/bin/python` | 3.11.15 / torch 2.5.1+cu124 | masks (SAM2.1-hiera-L video propagation), mask utilities |20| `/lp-dev/jonghoon/sam3/envs/sam3/bin/python` | 3.12.13 / torch 2.10.0+cu128 | masks (SAM3 text-prompted reference masks + text-prompted video propagation) |21| `/lp-dev/jonghoon/mimicgen_augment/envs/mimicgen/bin/python` | 3.10.20 / torch 2.5.1+cu121, **mujoco 3.3.1**, trimesh 4.12, coacd, imageio-ffmpeg | scene build, all MuJoCo rendering (`render_cam.py`), renderback compositing, simfix/seatfix/meshfix42/scaleup drivers, video encodes |22| `/data/nvidia/gripper_augmentator/conda-envs/mygr00t/bin/python` | 3.10.20 / torch 2.5.1+cu124 | GR00T-N1.5 policy inference (rollout, sweep, bref, rlwtwin). `mujoco==3.3.1` was side-installed with `pip install --target ws-root/work/rollout/pylibs` (NumPy + typing_extensions removed from the target) so one process holds policy + EGL renderer — see `05_rollout/README.md` |23| `/lp-dev/jonghoon/cosmos_predict25/envs/cosmos25/bin/python` | 3.10.20 / torch 2.7.0+cu128, transformer_engine 2.2 (unverified minor) | Cosmos-Predict1 SDEdit (`rb05/rb19/rb23`). NOTE: the *other* cosmos25 env under `/lp-dev/jonghoon/sandbox` lacks transformer_engine and does NOT work |24| `envs/void/bin/python` (project venv) | 3.10.20 / torch 2.7.0+cu128 | Netflix VOID (CogVideoX-Fun) ego inpainting |25| `envs/vace/bin/python` (project venv) | 3.10.20 / torch 2.7.0+cu128 | Wan2.1-VACE-14B-diffusers pilot |26| `envs/wan22/bin/python` (project venv) | 3.10.20 / torch 2.7.0+cu128 | Wan2.2-VACE-Fun-A14B pilot (via VideoX-Fun) |27| `envs/ltx2/bin/python` (project venv) | 3.13.13 / torch 2.13.0+cu132 | LTX-2 in/outpainting pilot |28| `/home/nvidia/miniconda3/envs/robocasa/bin/python` | (unverified) | glue/driver scripts, checkpoint downloads (`scripts/download_cosmos_ckpts.py`) |29 30## 2. Third-party repos (documented, NOT vendored)31 32Cloned under `/lp-dev/jonghoon/rebuttal/sim_action_aug/ext/` unless noted.33Commits read from the live clones.34 35| repo | remote | commit |36|---|---|---|37| Netflix VOID (`ext/void-model`) | https://github.com/Netflix/void-model.git | `e3914f8f551dd4b880661991fd6b28cd1699a97a` |38| GEN3C (`ext/GEN3C`) | https://github.com/nv-tlabs/GEN3C | `db2ffe12ced12ddafcec5e0422ee46ce8520746b` |39| StereoCrafter2 (`ext/StereoCrafter2`) | https://github.com/TencentARC/StereoCrafter.git | `6c6c700dbda139ee10a69a311b7d3a2a74ce165e` (research-only license) |40| SVOR (`ext/SVOR`) | https://github.com/xiaomi-research/svor | `df1fe23248c46477aea665c0f116fff91184f26d` |41| LTX-2 (`ext/LTX-2`) | https://github.com/Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` |42| ComfyUI-LTXVideo (`ext/ComfyUI-LTXVideo`) | https://github.com/Lightricks/ComfyUI-LTXVideo | `ac4d99839020b983e956a8ab67ec38aec1b6e65a` |43| VideoX-Fun (`ext/VideoX-Fun`) | https://github.com/aigc-apps/VideoX-Fun.git | `248ab0ac0ebc48f0b4ae43ceb2d7ded24cc907bb` |44| RobotSeg (`/lp-dev/jonghoon/rebuttal/ext_robotseg/RobotSeg`) | https://github.com/showlab/RobotSeg.git | `dafb8c0d507276e2f96d2b07ac3661a7b3a41a5f` |45| mujoco_menagerie (`/lp-dev/jonghoon/mujoco_menagerie`) | google-deepmind/mujoco_menagerie (unverified commit) | `franka_emika_panda/panda_nohand.xml`, `robotiq_2f85_v4/2f85.xml` |46| cosmos_sdedit (`/lp-dev/jonghoon/cosmos_sdedit`) | internal/local wrapper of NVIDIA Cosmos-Predict1 (unverified remote) | import-only use (`CosmosSDEditPipeline`) |47| SAM3D repo + `sam3d_hoi` driver | vendored inside `/data/nvidia/gripper_augmentator` (unverified remote) | used through `03_recon/r00_utils3d_shim.py` |48 49## 3. Model checkpoints50 51| model | source | note |52|---|---|---|53| GR00T-N1.5 finetune, **checkpoint-20000** | `/lp-dev/jonghoon/myGR00T_outputs/rebuttal_franka_eef80clean_new_emb_gbs32_20k_save5k_20260726/checkpoint-20000` | policy for the migrator pilot + scale-up (data config `franka_real_eef`, embodiment `new_embodiment`, registered in `myGR00T/gr00t/experiment/data_config.py`) |54| GR00T-N1.5 finetune, **checkpoint-15000** | same output dir (`SAA_CKPT`) | policy for the 80-episode rlwrld stage-1 run (see `14_rlwtwin/AUG_ACTIONS_STAGE1_REPORT.md`) |55| Cosmos-Predict1-7B-Text2World | HF `nvidia/Cosmos-Predict1-7B-Text2World` (gated — accept NVIDIA Open Model License) | SDEdit backbone; downloaded by `scripts/download_cosmos_ckpts.py` (58 GB incl. tokenizer + T5) |56| Cosmos-Tokenize1-CV8x8x8-720p | HF `nvidia/Cosmos-Tokenize1-CV8x8x8-720p` | |57| T5-11B | HF `google-t5/t5-11b` | |58| CogVideoX-Fun-V1.5-5b-InP | HF `alibaba-pai/CogVideoX-Fun-V1.5-5b-InP` | VOID base model |59| VOID LoRA passes | `void_pass1.safetensors` / `void_pass2.safetensors` from the Netflix void-model release | pass 2 tested and rejected |60| Wan2.1-VACE-14B-diffusers | HF (~70 GB) | pilot only |61| Wan2.2-VACE-Fun-A14B | HF `alibaba-pai/Wan2.2-VACE-Fun-A14B` | pilot only, rejected |62| GEN3C weights | NVIDIA release (~29 GB, NVIDIA Open Model License) | pilot only |63| SAM3 | `/lp-dev/jonghoon/sam3/checkpoints/sam3.pt` | |64| SAM2.1-hiera-L | `/data/nvidia/gripper_augmentator/checkpoints/sam2/sam2.1_hiera_large.pt` + cfg `configs/sam2.1/sam2.1_hiera_l.yaml` | |65| SAM3D-Objects checkpoints | inside `/data/nvidia/gripper_augmentator` (sam3d env) | scene reconstruction |66| Daimon DM-Tac W2M STEP CAD | dmrobot.com/service.html (free download) | superseded by AnySkin |67| AnySkin CAD | any-skin.github.io (NYU) project Google Drive | final exo tactile-fingertip mesh |68 69## 4. Simulation / rendering70 71- MuJoCo **3.3.1**, EGL rendering. `MUJOCO_EGL_DEVICE_ID` is NOT the nvidia-smi72  index — on the original box the mapping (read from `eglQueryDevicesEXT`) was73  EGL {0..7} → CUDA {2,3,0,1,6,7,4,5}. Every driver script sets both74  `CUDA_VISIBLE_DEVICES` and `MUJOCO_EGL_DEVICE_ID`; re-derive the mapping on any75  new machine.76- ZED SDK / `pyzed` for SVO depth extraction; the sam3d env's pyzed needs the77  TensorRT8/cuDNN8 `LD_LIBRARY_PATH` shim wrapped by `01_prep/p02_extract_depth.sh`.78- `HF_HOME` was pointed at the project (`sim_action_aug/hf_cache`), with79  `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` for policy runs.80- ffmpeg (libx264) — all encodes are detached CPU-only processes81  (`CUDA_VISIBLE_DEVICES=""`), a binding project rule.82 83## 5. Read-only data inputs (not copied)84 85- Migrator episodes: `/home/nvidia/jonghoon/rebuttal/migrator_data/episodes` (task86  `samdasoo_to_white_box_higher`, episodes 0004, 0007–0016) + its OLD authoritative87  `camera_calibration.json` and the solved `calibration_solved.json`.88- Swap meshes: `/home/nvidia/jonghoon/rebuttal/mesh/<object>/mesh_metric.glb`89  (abc_chocolate, green_mogi, green_pear, kanu, mango, orange_mogi, red_cleanser,90  white_cleanser). WARNING: the sibling `material_0.png` files have a mismatched91  atlas — always rebuild textures from the GLB's own `baseColorTexture`92  (`07_simfix/f01_fix_texture.py`, `13_scaleup/u01_mesh.py`).93- rlwrld 80-episode LeRobot export:94  `/lp-dev/jonghoon/rebuttal_data/lerobot_eef/franka_rebuttal_eef_80eps_clean`.95- Refined 80-ep extrinsic: `14_rlwtwin/calib80_board/out/calib_final.json`96  (copy included; originally `gripper_endpoint/calib80_board/out/`).97