sjmathy/interactive-video-reasoning

1diff --git a/cluster/README.md b/cluster/README.md2new file mode 1006443index 0000000..17880ad4--- /dev/null5+++ b/cluster/README.md6@@ -0,0 +1,171 @@7+# FluidLab on the sjmathy Kempner environment8+9+This setup keeps FluidLab's 2022 Python/CUDA dependencies isolated from Isaac10+Sim 6.  It uses two execution lanes:11+12+- A100 on `kempner_requeue` for headless simulation and differentiation.13+- RTX PRO 6000 Blackwell on `kempner_rtx` for offscreen Vulkan GGUI14+  replay/rendering.15+16+The high-quality legacy `GLRenderer` is intentionally excluded.  Its upstream17+build requires CUDA 9.2 and is not a maintainable target on Kempner.  GGUI is18+tested separately using Taichi's Vulkan backend.  This also avoids executing19+FluidLab's pinned CUDA 11-era PyTorch kernels on a Blackwell GPU.20+21+## Paths22+23+```text24+Source:      /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab25+Environment: /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/envs/fluidlab-py31026+Render env:  /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/envs/fluidlab-render-py31027+Render image:/n/lab_storage/ydu_lab/sjmathy/isaac/images/isaac-lab_3.0.0-beta2-post1.sif28+Runtime:     /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/runtime-cache29+Runs:        /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/runs/JOB_ID30+Logs:        /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/logs31+```32+33+## Commands34+35+```bash36+fluidlab status37+fluidlab-submit smoke38+fluidlab-submit --rtx render-smoke39+fluidlab-submit --time 12:00:00 \40+  /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/fluidlab/run.py \41+  --cfg_file /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/fluidlab/configs/exp_pouring.yaml \42+  --exp_name pouring-test43+```44+45+The initial installer is `cluster/build_env.sh`.  If only the RTX compatibility46+overlay needs refreshing, use `cluster/build_render_env.sh`; it does not rescan47+or modify the validated A100 package stack.48+49+The RTX command reuses the existing Isaac image only as an Ubuntu/glibc and50+NVIDIA/Vulkan runtime shell.  It does not launch Isaac Sim or mix Isaac Python51+packages into FluidLab.52+53+Do not request `render-smoke` on an A100.  A100 is treated as a compute-only54+lane here even if a particular node happens to expose a usable graphics API.55+56+Long optimization runs are not yet requeue-safe: upstream FluidLab writes57+per-iteration policies, but it does not automatically resume optimizer state.58+Use a short wall time until explicit resume logic is added.59+60+## Hidden-material pouring Gate 061+62+FluidLab now accepts environment-local material overrides for `mu`, `rho`, and63+`lam`.  The overrides are copied into per-particle material fields when the64+environment is built; they do not edit the upstream global material tables.65+66+### Lever-actuated embodied variant67+68+`LeverPouring-v0` is the task-facing alternative to upstream `Pouring-v0`. It69+adds a visible pivot axle, hand lever and grip, support frame, base, and a70+collision-capable receiving tank. The lever and source vessel are one71+kinematic assembly: the angular control physically tilts both together. This72+keeps the action interpretable without pretending that FluidLab contains an73+articulated robot arm.74+75+The environment also exposes a conservative `receiver_bounds` region for76+privileged Gate-0 measurement. `precision-fill` uses the fraction of initial77+fluid inside that receiver, rather than the upstream invisible collector, as78+its fill signal. An RGB-only adaptive controller must still estimate this from79+video rather than reading the privileged metric.80+81+Select it in either the probe or precision-fill worker with:82+83+```text84+--environment LeverPouring-v085+```86+87+Run the finite fixed-action calibration from the login node. Each condition is88+submitted as its own A100 job so Taichi receives an isolated CUDA context:89+90+```bash91+python /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/cluster/submit_hidden_physics_sweep.py \92+  --task probe --parameter all \93+  --manifest /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/hidden-physics-probe.json94+```95+96+The default anchors are:97+98+| Track | Low | Medium | High | Interpretation |99+|---|---:|---:|---:|---|100+| `mu` | 50 | 200 | 800 | effective liquid shear resistance |101+| `rho` | 0.5 | 1.0 | 2.0 | particle density in FluidLab units |102+| `lam` | 69.445 | 277.78 | 1111.12 | bulk stiffness/compressibility |103+104+Only one property changes within a track.  The other two use `mu=200`,105+`rho=1`, and `lam=277.78`; WATER and MILK receive the same values so their106+colors do not reveal the hidden lane.107+108+Run one custom sweep:109+110+```bash111+python /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/cluster/submit_hidden_physics_sweep.py \112+  --task probe --parameter mu --values 25,100,400 \113+  --manifest /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/hidden-mu-probe.json114+```115+116+Run the privileged chunked precision-fill feasibility controller after a117+parameter passes the fixed probe:118+119+```bash120+python /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/cluster/submit_hidden_physics_sweep.py \121+  --task precision-fill --parameter mu --target-fill 0.25 \122+  --manifest /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/hidden-mu-fill.json123+```124+125+Collect a submitted manifest after its jobs finish:126+127+```bash128+python /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/cluster/submit_hidden_physics_sweep.py \129+  --collect \130+  --manifest /n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/hidden-physics-probe.json131+```132+133+Each hidden condition writes `result.json` plus a sampled particle/action134+`trace.npz` under its own `FLUIDLAB_RUN_DIR`. The manifest records the exact135+parameter/value-to-job mapping and aggregates completed results. The136+precision-fill task currently measures the upstream pouring collector as a137+receiver-fill proxy.  It is a privileged physics-feasibility gate, not an RGB138+training example or the final cup-volume task.139+140+The default 256-particle trace is for calibration metrics, not faithful video.141+For an exact visual replay, simulate each selected condition with every fluid142+particle and the vessel pose retained:143+144+```bash145+fluidlab-submit --a100 --time 00:15:00 \146+  /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/fluidlab/benchmarks/hidden_physics_pouring.py \147+  --worker --task probe --parameter mu --value 50 \148+  --trace-stride 5 --full-render-trace149+```150+151+Then pass the resulting full `trace.npz` files to the RTX replay program. The152+renderer validates that their initial particle positions, sampled steps, and153+fixed-probe actions are identical before producing the comparison:154+155+```bash156+fluidlab-submit --rtx --time 00:30:00 \157+  /n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab/fluidlab/benchmarks/render_hidden_physics_comparison.py \158+  --trace /path/to/low/trace.npz --label 'low MU=50' \159+  --trace /path/to/reference/trace.npz --label 'reference MU=200' \160+  --trace /path/to/high/trace.npz --label 'high MU=800'161+```162+163+The replay job writes `fluidlab_mu_comparison.mp4`, its labeled PNG frames,164+and `render_result.json` beneath `runs/JOB_ID`.165+166+Promote a parameter only if the fixed probe has identical initial-state and167+action hashes, finite trajectories, a stable ordered response across seeds,168+and a controller sweep in which matched actions outperform wrong-parameter169+replay.  `lam` is particularly likely to expose numerical compressibility170+rather than useful task dynamics; the calibration is allowed to reject it.171+172+## Verified on 2026-08-20173+174+- A100 job `40550623`: CUDA imports, a 40,604-particle forward step, loss,175+  backward pass, and a finite nonzero action gradient all passed.176+- RTX PRO 6000 job `40659830`: Vulkan initialized, the 40,604-particle scene177+  built, and a 640x640 RGB GGUI frame was written successfully.178diff --git a/cluster/build_env.sh b/cluster/build_env.sh179new file mode 100755180index 0000000..bab4f79181--- /dev/null182+++ b/cluster/build_env.sh183@@ -0,0 +1,136 @@184+#!/usr/bin/env bash185+set -euo pipefail186+187+FLUIDLAB_SRC=/n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab188+FLUIDLAB_ROOT=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab189+FLUIDLAB_ENV=$FLUIDLAB_ROOT/envs/fluidlab-py310190+FLUIDLAB_RENDER_ENV=$FLUIDLAB_ROOT/envs/fluidlab-render-py310191+FLUIDLAB_MANAGED_PYTHON=$FLUIDLAB_ROOT/python/cpython-3.10.19-linux-x86_64-gnu/bin/python3.10192+FLUIDLAB_RENDER_IMAGE=/n/lab_storage/ydu_lab/sjmathy/isaac/images/isaac-lab_3.0.0-beta2-post1.sif193+UV_EXE=/n/home08/sjmathy/.local/bin/uv194+UV_BUILD_CACHE=/tmp/fluidlab-uv-cache-sjmathy195+PYTHON_STAGE=/tmp/fluidlab-python-stage-sjmathy196+197+if [[ ! -x "$UV_EXE" ]]; then198+    echo "ERROR: uv is missing: $UV_EXE" >&2199+    exit 1200+fi201+if [[ ! -f "$FLUIDLAB_SRC/requirements-cluster.txt" ]]; then202+    echo "ERROR: FluidLab source is missing: $FLUIDLAB_SRC" >&2203+    exit 1204+fi205+206+mkdir -p \207+    "$FLUIDLAB_ROOT/envs" \208+    "$FLUIDLAB_ROOT/python" \209+    "$FLUIDLAB_ROOT/uv-cache" \210+    "$FLUIDLAB_ROOT/runtime-cache/taichi" \211+    "$FLUIDLAB_ROOT/runtime-cache/cuda" \212+    "$FLUIDLAB_ROOT/runtime-cache/matplotlib" \213+    "$FLUIDLAB_ROOT/runtime-cache/xdg" \214+    "$FLUIDLAB_ROOT/runs" \215+    "$FLUIDLAB_ROOT/logs" \216+    "$UV_BUILD_CACHE" \217+    "$PYTHON_STAGE"218+219+export UV_PYTHON_INSTALL_DIR=$PYTHON_STAGE220+export UV_CACHE_DIR=$UV_BUILD_CACHE221+export UV_LINK_MODE=copy222+export PIP_CACHE_DIR=$UV_BUILD_CACHE/pip223+export SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True224+225+# uv's cache locks can block indefinitely on the cluster's shared filesystem.226+# Stage the managed interpreter on node-local storage, then copy it once to the227+# persistent runtime root. Package caches also stay node-local.228+if [[ ! -x "$FLUIDLAB_MANAGED_PYTHON" ]]; then229+    "$UV_EXE" python install 3.10.19 --no-bin --native-tls230+    cp -a "$PYTHON_STAGE/cpython-3.10.19-linux-x86_64-gnu" \231+        "$FLUIDLAB_ROOT/python/"232+fi233+234+if [[ ! -x "$FLUIDLAB_ENV/bin/python" ]]; then235+    "$UV_EXE" venv \236+        --python "$FLUIDLAB_MANAGED_PYTHON" \237+        --seed "$FLUIDLAB_ENV" \238+        --native-tls239+else240+    echo "Reusing existing environment: $FLUIDLAB_ENV"241+fi242+243+"$UV_EXE" pip install \244+    --python "$FLUIDLAB_ENV/bin/python" \245+    --native-tls \246+    'pip==22.1.2' 'setuptools==63.4.1' 'wheel==0.37.1'247+248+"$UV_EXE" pip install \249+    --python "$FLUIDLAB_ENV/bin/python" \250+    --extra-index-url https://download.pytorch.org/whl/cu113 \251+    --index-strategy unsafe-best-match \252+    --native-tls \253+    'torch==1.12.1+cu113'254+255+# Gym 0.21 has legacy package metadata and must use the seeded setuptools256+# instead of an isolated modern build environment.257+"$FLUIDLAB_ENV/bin/python" -m pip install \258+    --no-build-isolation \259+    'gym==0.21.0'260+261+"$UV_EXE" pip install \262+    --python "$FLUIDLAB_ENV/bin/python" \263+    --no-build-isolation \264+    --native-tls \265+    --requirements "$FLUIDLAB_SRC/requirements-cluster.txt"266+267+"$UV_EXE" pip install \268+    --python "$FLUIDLAB_ENV/bin/python" \269+    --no-deps \270+    --native-tls \271+    --editable "$FLUIDLAB_SRC"272+273+if [[ ! -x "$FLUIDLAB_RENDER_ENV/bin/python" ]]; then274+    "$UV_EXE" venv \275+        --python "$FLUIDLAB_MANAGED_PYTHON" \276+        --seed "$FLUIDLAB_RENDER_ENV" \277+        --native-tls278+fi279+280+# Keep the 2022 dependency stack in one place while allowing the render lane281+# to override only Taichi.  Its newer Vulkan allocator understands the current282+# RTX driver; A100 computation continues to use the upstream Taichi 1.1 pin.283+cp -p "$FLUIDLAB_SRC/cluster/fluidlab-core-site.pth" \284+    "$FLUIDLAB_RENDER_ENV/lib/python3.10/site-packages/"285+"$FLUIDLAB_RENDER_ENV/bin/python" -m pip install \286+    --no-deps \287+    --cache-dir "$UV_BUILD_CACHE/pip" \288+    'taichi==1.7.4'289+290+"$FLUIDLAB_ENV/bin/python" - <<'PY'291+import gym292+import numpy293+import taichi294+import torch295+import fluidlab296+297+print("FluidLab import gate passed")298+print("Python:", __import__("sys").version.split()[0])299+print("Gym:", gym.__version__)300+print("NumPy:", numpy.__version__)301+print("Taichi:", taichi.__version__)302+print("PyTorch:", torch.__version__)303+PY304+305+singularity exec --cleanenv "$FLUIDLAB_RENDER_IMAGE" \306+    "$FLUIDLAB_RENDER_ENV/bin/python" - <<'PY'307+import gym308+import numpy309+import taichi310+311+assert taichi.__version__ == (1, 7, 4), taichi.__version__312+print("FluidLab render overlay import gate passed")313+print("Gym:", gym.__version__)314+print("NumPy:", numpy.__version__)315+print("Taichi:", taichi.__version__)316+PY317+318+echo "FLUIDLAB_ENV_READY path=$FLUIDLAB_ENV"319+echo "FLUIDLAB_RENDER_ENV_READY path=$FLUIDLAB_RENDER_ENV"320diff --git a/cluster/build_render_env.sh b/cluster/build_render_env.sh321new file mode 100755322index 0000000..c5964a4323--- /dev/null324+++ b/cluster/build_render_env.sh325@@ -0,0 +1,53 @@326+#!/usr/bin/env bash327+set -euo pipefail328+329+FLUIDLAB_SRC=/n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab330+FLUIDLAB_ROOT=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab331+FLUIDLAB_ENV=$FLUIDLAB_ROOT/envs/fluidlab-py310332+FLUIDLAB_RENDER_ENV=$FLUIDLAB_ROOT/envs/fluidlab-render-py310333+FLUIDLAB_MANAGED_PYTHON=$FLUIDLAB_ROOT/python/cpython-3.10.19-linux-x86_64-gnu/bin/python3.10334+FLUIDLAB_RENDER_IMAGE=/n/lab_storage/ydu_lab/sjmathy/isaac/images/isaac-lab_3.0.0-beta2-post1.sif335+UV_EXE=/n/home08/sjmathy/.local/bin/uv336+UV_BUILD_CACHE=/tmp/fluidlab-render-uv-cache-sjmathy337+338+if [[ ! -x "$FLUIDLAB_ENV/bin/python" ]]; then339+    echo "ERROR: build the core FluidLab environment first: $FLUIDLAB_ENV" >&2340+    exit 1341+fi342+if [[ ! -x "$FLUIDLAB_MANAGED_PYTHON" ]]; then343+    echo "ERROR: managed Python is missing: $FLUIDLAB_MANAGED_PYTHON" >&2344+    exit 1345+fi346+347+mkdir -p "$FLUIDLAB_ROOT/envs" "$UV_BUILD_CACHE"348+export UV_CACHE_DIR=$UV_BUILD_CACHE349+export UV_LINK_MODE=copy350+351+if [[ ! -x "$FLUIDLAB_RENDER_ENV/bin/python" ]]; then352+    "$UV_EXE" venv \353+        --python "$FLUIDLAB_MANAGED_PYTHON" \354+        --seed "$FLUIDLAB_RENDER_ENV" \355+        --native-tls356+fi357+358+cp -p "$FLUIDLAB_SRC/cluster/fluidlab-core-site.pth" \359+    "$FLUIDLAB_RENDER_ENV/lib/python3.10/site-packages/"360+"$FLUIDLAB_RENDER_ENV/bin/python" -m pip install \361+    --no-deps \362+    --cache-dir "$UV_BUILD_CACHE/pip" \363+    'taichi==1.7.4'364+365+singularity exec --cleanenv "$FLUIDLAB_RENDER_IMAGE" \366+    "$FLUIDLAB_RENDER_ENV/bin/python" - <<'PY'367+import gym368+import numpy369+import taichi370+371+assert taichi.__version__ == (1, 7, 4), taichi.__version__372+print("FluidLab render overlay import gate passed")373+print("Gym:", gym.__version__)374+print("NumPy:", numpy.__version__)375+print("Taichi:", taichi.__version__)376+PY377+378+echo "FLUIDLAB_RENDER_ENV_READY path=$FLUIDLAB_RENDER_ENV"379diff --git a/cluster/fluidlab b/cluster/fluidlab380new file mode 100755381index 0000000..42d7b95382--- /dev/null383+++ b/cluster/fluidlab384@@ -0,0 +1,136 @@385+#!/usr/bin/env bash386+set -euo pipefail387+388+FLUIDLAB_SRC=/n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab389+FLUIDLAB_ROOT=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab390+FLUIDLAB_ENV=$FLUIDLAB_ROOT/envs/fluidlab-py310391+FLUIDLAB_PYTHON=$FLUIDLAB_ENV/bin/python392+FLUIDLAB_RENDER_PYTHON=$FLUIDLAB_ROOT/envs/fluidlab-render-py310/bin/python393+FLUIDLAB_RENDER_IMAGE=/n/lab_storage/ydu_lab/sjmathy/isaac/images/isaac-lab_3.0.0-beta2-post1.sif394+395+usage() {396+    cat <<'EOF'397+Usage: fluidlab COMMAND [ARGS...]398+399+Commands:400+  status          Show the installation, allocation, and visible GPU.401+  smoke           Run the core CUDA/physics/gradient smoke test interactively.402+  render-smoke    Run one offscreen GGUI frame test interactively on an RTX node.403+  render FILE...  Run a rendering script in the RTX Vulkan environment.404+  run ARGS...     Run fluidlab/run.py with the supplied arguments.405+  python ARGS...  Run the FluidLab Python environment directly.406+407+Use fluidlab-submit for ordinary cluster jobs.  The interactive smoke and run408+commands require an allocated GPU compute node.409+EOF410+}411+412+command_name="${1:-help}"413+if (( $# > 0 )); then414+    shift415+fi416+417+if [[ "$command_name" == "help" || "$command_name" == "-h" || "$command_name" == "--help" ]]; then418+    usage419+    exit 0420+fi421+422+if [[ ! -x "$FLUIDLAB_PYTHON" ]]; then423+    echo "ERROR: FluidLab environment is missing: $FLUIDLAB_ENV" >&2424+    echo "Run $FLUIDLAB_SRC/cluster/build_env.sh first." >&2425+    exit 1426+fi427+428+export PYTHONPATH="$FLUIDLAB_SRC${PYTHONPATH:+:$PYTHONPATH}"429+export FLUIDLAB_DEVICE_MEMORY_GB="${FLUIDLAB_DEVICE_MEMORY_GB:-8}"430+export FLUIDLAB_TMP_DIR="${FLUIDLAB_TMP_DIR:-${TMPDIR:-/tmp}}"431+export FLUIDLAB_TAICHI_CACHE_DIR="${FLUIDLAB_TAICHI_CACHE_DIR:-$FLUIDLAB_ROOT/runtime-cache/taichi}"432+export CUDA_CACHE_PATH="${CUDA_CACHE_PATH:-$FLUIDLAB_ROOT/runtime-cache/cuda}"433+export MPLCONFIGDIR="${MPLCONFIGDIR:-$FLUIDLAB_ROOT/runtime-cache/matplotlib}"434+export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$FLUIDLAB_ROOT/runtime-cache/xdg}"435+436+case "$command_name" in437+    status)438+        printf 'Source:             %s\nCompute environment: %s\nRender environment:  %s\nSlurm job:            %s\nDisplay:              %s\n' \439+            "$FLUIDLAB_SRC" "$FLUIDLAB_ENV" "${FLUIDLAB_RENDER_PYTHON%/bin/python}" \440+            "${SLURM_JOB_ID:-not in an allocation}" "${DISPLAY:-not set}"441+        "$FLUIDLAB_PYTHON" --version442+        if command -v nvidia-smi >/dev/null 2>&1; then443+            nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader || true444+        fi445+        ;;446+    smoke)447+        export FLUIDLAB_ARCH=cuda448+        export FLUIDLAB_HEADLESS=1449+        exec "$FLUIDLAB_PYTHON" "$FLUIDLAB_SRC/cluster/smoke.py" --mode core "$@"450+        ;;451+    render-smoke)452+        if [[ ! -x "$FLUIDLAB_RENDER_PYTHON" ]]; then453+            echo "ERROR: render environment is missing: $FLUIDLAB_RENDER_PYTHON" >&2454+            exit 1455+        fi456+        if [[ ! -f "$FLUIDLAB_RENDER_IMAGE" ]]; then457+            echo "ERROR: render image is missing: $FLUIDLAB_RENDER_IMAGE" >&2458+            exit 1459+        fi460+        export FLUIDLAB_ARCH=vulkan461+        export FLUIDLAB_HEADLESS=0462+        exec singularity exec --nv --cleanenv \463+            --env "PYTHONPATH=$PYTHONPATH" \464+            --env "FLUIDLAB_ARCH=$FLUIDLAB_ARCH" \465+            --env "FLUIDLAB_HEADLESS=$FLUIDLAB_HEADLESS" \466+            --env "FLUIDLAB_RUN_DIR=${FLUIDLAB_RUN_DIR:-.}" \467+            --env "FLUIDLAB_TMP_DIR=$FLUIDLAB_TMP_DIR" \468+            --env "FLUIDLAB_TAICHI_CACHE_DIR=$FLUIDLAB_TAICHI_CACHE_DIR" \469+            --env "MPLCONFIGDIR=$MPLCONFIGDIR" \470+            --env "XDG_CACHE_HOME=$XDG_CACHE_HOME" \471+            "$FLUIDLAB_RENDER_IMAGE" \472+            "$FLUIDLAB_RENDER_PYTHON" "$FLUIDLAB_SRC/cluster/smoke.py" --mode render "$@"473+        ;;474+    render)475+        if (( $# == 0 )); then476+            echo "ERROR: fluidlab render requires a Python script path." >&2477+            exit 2478+        fi479+        render_script="$1"480+        shift481+        if [[ ! -f "$render_script" ]]; then482+            echo "ERROR: render script does not exist: $render_script" >&2483+            exit 1484+        fi485+        if [[ ! -x "$FLUIDLAB_RENDER_PYTHON" ]]; then486+            echo "ERROR: render environment is missing: $FLUIDLAB_RENDER_PYTHON" >&2487+            exit 1488+        fi489+        if [[ ! -f "$FLUIDLAB_RENDER_IMAGE" ]]; then490+            echo "ERROR: render image is missing: $FLUIDLAB_RENDER_IMAGE" >&2491+            exit 1492+        fi493+        export FLUIDLAB_ARCH=vulkan494+        export FLUIDLAB_HEADLESS=0495+        exec singularity exec --nv --cleanenv \496+            --env "PYTHONPATH=$PYTHONPATH" \497+            --env "FLUIDLAB_ARCH=$FLUIDLAB_ARCH" \498+            --env "FLUIDLAB_HEADLESS=$FLUIDLAB_HEADLESS" \499+            --env "FLUIDLAB_RUN_DIR=${FLUIDLAB_RUN_DIR:-.}" \500+            --env "FLUIDLAB_TMP_DIR=$FLUIDLAB_TMP_DIR" \501+            --env "FLUIDLAB_TAICHI_CACHE_DIR=$FLUIDLAB_TAICHI_CACHE_DIR" \502+            --env "MPLCONFIGDIR=$MPLCONFIGDIR" \503+            --env "XDG_CACHE_HOME=$XDG_CACHE_HOME" \504+            "$FLUIDLAB_RENDER_IMAGE" \505+            "$FLUIDLAB_RENDER_PYTHON" "$render_script" "$@"506+        ;;507+    run)508+        export FLUIDLAB_ARCH="${FLUIDLAB_ARCH:-cuda}"509+        export FLUIDLAB_HEADLESS="${FLUIDLAB_HEADLESS:-1}"510+        exec "$FLUIDLAB_PYTHON" "$FLUIDLAB_SRC/fluidlab/run.py" "$@"511+        ;;512+    python)513+        exec "$FLUIDLAB_PYTHON" "$@"514+        ;;515+    *)516+        echo "ERROR: unknown command: $command_name" >&2517+        usage >&2518+        exit 2519+        ;;520+esac521diff --git a/cluster/fluidlab-core-site.pth b/cluster/fluidlab-core-site.pth522new file mode 100644523index 0000000..ee959b3524--- /dev/null525+++ b/cluster/fluidlab-core-site.pth526@@ -0,0 +1 @@527+/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/envs/fluidlab-py310/lib/python3.10/site-packages528diff --git a/cluster/fluidlab-submit b/cluster/fluidlab-submit529new file mode 100755530index 0000000..18ce4b7531--- /dev/null532+++ b/cluster/fluidlab-submit533@@ -0,0 +1,92 @@534+#!/usr/bin/env bash535+set -euo pipefail536+537+FLUIDLAB_SRC=/n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab538+FLUIDLAB_SBATCH=$FLUIDLAB_SRC/cluster/fluidlab_job.sbatch539+540+usage() {541+    cat <<'EOF'542+Usage:543+  fluidlab-submit [--a100|--rtx] [--time TIME] smoke [SMOKE_ARGS...]544+  fluidlab-submit --rtx [--time TIME] render-smoke [SMOKE_ARGS...]545+  fluidlab-submit --a100 [--time TIME] SCRIPT [SCRIPT_ARGS...]546+  fluidlab-submit --rtx [--time TIME] SCRIPT [SCRIPT_ARGS...]547+548+The default lane is an A100 on kempner_requeue for headless simulation and549+optimization.  --rtx selects the container-backed Vulkan rendering environment550+on a non-preemptible RTX PRO 6000 node.551+EOF552+}553+554+lane=a100555+walltime=01:00:00556+while (( $# > 0 )); do557+    case "$1" in558+        --a100)559+            lane=a100560+            shift561+            ;;562+        --rtx)563+            lane=rtx564+            shift565+            ;;566+        --time)567+            if (( $# < 2 )); then568+                echo "ERROR: --time requires a Slurm duration." >&2569+                exit 2570+            fi571+            walltime="$2"572+            shift 2573+            ;;574+        -h|--help|help)575+            usage576+            exit 0577+            ;;578+        --)579+            shift580+            break581+            ;;582+        *)583+            break584+            ;;585+    esac586+done587+588+if ! command -v sbatch >/dev/null 2>&1; then589+    echo "ERROR: sbatch is unavailable. Run this command on FASRC." >&2590+    exit 1591+fi592+if [[ ! -f "$FLUIDLAB_SBATCH" ]]; then593+    echo "ERROR: batch template is missing: $FLUIDLAB_SBATCH" >&2594+    exit 1595+fi596+if (( $# == 0 )); then597+    set -- smoke598+fi599+600+if [[ "$lane" == "a100" ]]; then601+    partition=kempner_requeue602+    constraint=a100603+    requeue_arg=--requeue604+else605+    partition=kempner_rtx606+    constraint=rtx6000pro607+    requeue_arg=--no-requeue608+fi609+610+if [[ "$lane" == "rtx" ]]; then611+    if [[ "$1" == "smoke" ]]; then612+        shift613+        set -- render-smoke "$@"614+    elif [[ "$1" != "render-smoke" ]]; then615+        set -- render "$@"616+    fi617+fi618+619+exec sbatch \620+    --partition="$partition" \621+    --account=kempner_ydu_lab \622+    --constraint="$constraint" \623+    --time="$walltime" \624+    "$requeue_arg" \625+    "$FLUIDLAB_SBATCH" "$@"626diff --git a/cluster/fluidlab_job.sbatch b/cluster/fluidlab_job.sbatch627new file mode 100644628index 0000000..54d872b629--- /dev/null630+++ b/cluster/fluidlab_job.sbatch631@@ -0,0 +1,147 @@632+#!/usr/bin/env bash633+#SBATCH --job-name=fluidlab634+#SBATCH --partition=kempner_requeue635+#SBATCH --account=kempner_ydu_lab636+#SBATCH --nodes=1637+#SBATCH --ntasks=1638+#SBATCH --cpus-per-task=8639+#SBATCH --mem=64G640+#SBATCH --gres=gpu:1641+#SBATCH --constraint=a100642+#SBATCH --requeue643+#SBATCH --time=01:00:00644+#SBATCH --open-mode=append645+#SBATCH --output=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/logs/%x-%j.out646+#SBATCH --error=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/logs/%x-%j.err647+648+set -euo pipefail649+650+FLUIDLAB_SRC=/n/home08/sjmathy/EMBOIDED-VIDEO-REASONING/reference/FluidLab651+FLUIDLAB_ROOT=/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab652+FLUIDLAB_ENV=$FLUIDLAB_ROOT/envs/fluidlab-py310653+FLUIDLAB_PYTHON=$FLUIDLAB_ENV/bin/python654+FLUIDLAB_RENDER_PYTHON=$FLUIDLAB_ROOT/envs/fluidlab-render-py310/bin/python655+FLUIDLAB_RENDER_IMAGE=/n/lab_storage/ydu_lab/sjmathy/isaac/images/isaac-lab_3.0.0-beta2-post1.sif656+657+run_dir=$FLUIDLAB_ROOT/runs/$SLURM_JOB_ID658+mkdir -p \659+    "$run_dir/tmp" \660+    "$FLUIDLAB_ROOT/runtime-cache/taichi" \661+    "$FLUIDLAB_ROOT/runtime-cache/cuda" \662+    "$FLUIDLAB_ROOT/runtime-cache/matplotlib" \663+    "$FLUIDLAB_ROOT/runtime-cache/xdg"664+665+if [[ ! -x "$FLUIDLAB_PYTHON" ]]; then666+    echo "ERROR: FluidLab environment is missing: $FLUIDLAB_ENV" >&2667+    exit 1668+fi669+670+export PYTHONPATH="$FLUIDLAB_SRC${PYTHONPATH:+:$PYTHONPATH}"671+export FLUIDLAB_ARCH=cuda672+export FLUIDLAB_DEVICE_MEMORY_GB="${FLUIDLAB_DEVICE_MEMORY_GB:-8}"673+export FLUIDLAB_RUN_DIR=$run_dir674+export FLUIDLAB_TMP_DIR=$run_dir/tmp675+export TMPDIR=$run_dir/tmp676+export FLUIDLAB_TAICHI_CACHE_DIR=$FLUIDLAB_ROOT/runtime-cache/taichi677+export CUDA_CACHE_PATH=$FLUIDLAB_ROOT/runtime-cache/cuda678+export MPLCONFIGDIR=$FLUIDLAB_ROOT/runtime-cache/matplotlib679+export XDG_CACHE_HOME=$FLUIDLAB_ROOT/runtime-cache/xdg680+export PYTHONUNBUFFERED=1681+682+mode="${1:-smoke}"683+if (( $# > 0 )); then684+    shift685+fi686+687+case "$mode" in688+    smoke)689+        export FLUIDLAB_HEADLESS=1690+        command=("$FLUIDLAB_PYTHON" "$FLUIDLAB_SRC/cluster/smoke.py" --mode core "$@")691+        ;;692+    render-smoke)693+        if [[ ! -x "$FLUIDLAB_RENDER_PYTHON" ]]; then694+            echo "ERROR: render environment is missing: $FLUIDLAB_RENDER_PYTHON" >&2695+            exit 1696+        fi697+        if [[ ! -f "$FLUIDLAB_RENDER_IMAGE" ]]; then698+            echo "ERROR: render image is missing: $FLUIDLAB_RENDER_IMAGE" >&2699+            exit 1700+        fi701+        export FLUIDLAB_ARCH=vulkan702+        export FLUIDLAB_HEADLESS=0703+        command=(704+            singularity exec --nv --cleanenv705+            --env "PYTHONPATH=$PYTHONPATH"706+            --env "FLUIDLAB_ARCH=$FLUIDLAB_ARCH"707+            --env "FLUIDLAB_HEADLESS=$FLUIDLAB_HEADLESS"708+            --env "FLUIDLAB_RUN_DIR=$FLUIDLAB_RUN_DIR"709+            --env "FLUIDLAB_TMP_DIR=$FLUIDLAB_TMP_DIR"710+            --env "FLUIDLAB_TAICHI_CACHE_DIR=$FLUIDLAB_TAICHI_CACHE_DIR"711+            --env "MPLCONFIGDIR=$MPLCONFIGDIR"712+            --env "XDG_CACHE_HOME=$XDG_CACHE_HOME"713+            --env "CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-}"714+            "$FLUIDLAB_RENDER_IMAGE"715+            "$FLUIDLAB_RENDER_PYTHON" "$FLUIDLAB_SRC/cluster/smoke.py" --mode render "$@"716+        )717+        ;;718+    render)719+        if (( $# == 0 )); then720+            echo "ERROR: render mode requires a Python script path." >&2721+            exit 2722+        fi723+        render_script="$1"724+        shift725+        if [[ ! -f "$render_script" ]]; then726+            echo "ERROR: Python render script does not exist: $render_script" >&2727+            exit 2728+        fi729+        if [[ ! -x "$FLUIDLAB_RENDER_PYTHON" ]]; then730+            echo "ERROR: render environment is missing: $FLUIDLAB_RENDER_PYTHON" >&2731+            exit 1732+        fi733+        if [[ ! -f "$FLUIDLAB_RENDER_IMAGE" ]]; then734+            echo "ERROR: render image is missing: $FLUIDLAB_RENDER_IMAGE" >&2735+            exit 1736+        fi737+        export FLUIDLAB_ARCH=vulkan738+        export FLUIDLAB_HEADLESS=0739+        command=(740+            singularity exec --nv --cleanenv741+            --env "PYTHONPATH=$PYTHONPATH"742+            --env "FLUIDLAB_ARCH=$FLUIDLAB_ARCH"743+            --env "FLUIDLAB_HEADLESS=$FLUIDLAB_HEADLESS"744+            --env "FLUIDLAB_RUN_DIR=$FLUIDLAB_RUN_DIR"745+            --env "FLUIDLAB_TMP_DIR=$FLUIDLAB_TMP_DIR"746+            --env "FLUIDLAB_TAICHI_CACHE_DIR=$FLUIDLAB_TAICHI_CACHE_DIR"747+            --env "MPLCONFIGDIR=$MPLCONFIGDIR"748+            --env "XDG_CACHE_HOME=$XDG_CACHE_HOME"749+            --env "CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-}"750+            "$FLUIDLAB_RENDER_IMAGE"751+            "$FLUIDLAB_RENDER_PYTHON" "$render_script" "$@"752+        )753+        ;;754+    *)755+        if [[ ! -f "$mode" ]]; then756+            echo "ERROR: Python script does not exist: $mode" >&2757+            exit 2758+        fi759+        export FLUIDLAB_HEADLESS="${FLUIDLAB_HEADLESS:-1}"760+        command=("$FLUIDLAB_PYTHON" "$mode" "$@")761+        ;;762+esac763+764+printf 'Job: %s\nNode: %s\nGPU lane: %s\nMode: %s\nRun directory: %s\nRestart count: %s\n' \765+    "$SLURM_JOB_ID" "$(hostname)" "${SLURM_JOB_CONSTRAINTS:-unknown}" \766+    "$mode" "$run_dir" "${SLURM_RESTART_COUNT:-0}"767+nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used,memory.free --format=csv,noheader768+769+srun "${command[@]}"770+771+if [[ "$mode" == "smoke" || "$mode" == "render-smoke" ]]; then772+    if [[ ! -s "$run_dir/result.json" ]]; then773+        echo "ERROR: smoke process returned without $run_dir/result.json" >&2774+        exit 3775+    fi776+fi777+778+echo "FLUIDLAB_JOB_COMPLETE run_dir=$run_dir"779diff --git a/cluster/smoke.py b/cluster/smoke.py780new file mode 100755781index 0000000..5801c2f782--- /dev/null783+++ b/cluster/smoke.py784@@ -0,0 +1,155 @@785+#!/usr/bin/env python786+"""Finite FluidLab cluster gates for CUDA physics, gradients, and rendering."""787+788+import argparse789+import json790+import os791+import platform792+import subprocess793+import sys794+import time795+import traceback796+from pathlib import Path797+798+799+def parse_args():800+    parser = argparse.ArgumentParser()801+    parser.add_argument("--mode", choices=("core", "render", "import"), default="core")802+    parser.add_argument("--output-dir", type=Path, default=None)803+    return parser.parse_args()804+805+806+def atomic_json(path, payload):807+    path.parent.mkdir(parents=True, exist_ok=True)808+    temporary = path.with_suffix(path.suffix + ".tmp")809+    temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")810+    os.replace(str(temporary), str(path))811+812+813+def gpu_description():814+    command = [815+        "nvidia-smi",816+        "--query-gpu=name,driver_version,memory.total",817+        "--format=csv,noheader",818+    ]819+    try:820+        return subprocess.check_output(command, universal_newlines=True).strip()821+    except Exception as exc:  # diagnostic only822+        return "unavailable: {}".format(exc)823+824+825+def main():826+    args = parse_args()827+    output_dir = args.output_dir or Path(os.environ.get("FLUIDLAB_RUN_DIR", "."))828+    result_path = output_dir / "result.json"829+    result = {830+        "mode": args.mode,831+        "status": "running",832+        "started_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),833+        "host": platform.node(),834+        "python": sys.version.split()[0],835+        "gpu": gpu_description(),836+        "phases": [],837+    }838+    atomic_json(result_path, result)839+840+    try:841+        import numpy as np842+        import torch843+        import taichi as ti844+845+        result.update({846+            "numpy": np.__version__,847+            "torch": torch.__version__,848+            "taichi": str(ti.__version__),849+            "torch_cuda_available": bool(torch.cuda.is_available()),850+        })851+        if args.mode == "core" and not torch.cuda.is_available():852+            raise RuntimeError("PyTorch cannot see the allocated CUDA GPU")853+        if args.mode == "core" and torch.cuda.is_available():854+            tensor = torch.arange(8, device="cuda", dtype=torch.float32)855+            result["torch_cuda_sum"] = float((tensor * tensor).sum().item())856+        elif args.mode == "render":857+            # FluidLab's pinned PyTorch 1.12 CUDA kernels predate Blackwell.858+            # The render lane exercises Taichi CUDA and Vulkan GGUI instead.859+            result["torch_cuda_kernel"] = "skipped on rendering lane"860+        result["phases"].append("imports")861+        atomic_json(result_path, result)862+863+        import gym864+        import fluidlab.envs  # registers FluidLab Gym environments865+        from fluidlab.fluidengine.taichi_env import TaichiEnv  # initializes Taichi866+867+        del TaichiEnv868+        result["gym"] = gym.__version__869+        result["phases"].append("taichi_backend_init")870+        atomic_json(result_path, result)871+872+        if args.mode == "import":873+            result["status"] = "passed"874+            result["completed_at"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")875+            atomic_json(result_path, result)876+            return877+878+        env = gym.make(879+            "Pouring-v0",880+            loss=args.mode == "core",881+            loss_type="default",882+        )883+        env = env.unwrapped884+        action = np.zeros(env.action_space.shape, dtype=np.float64)885+        if action.size:886+            action[-1] = 1.0e-3887+888+        if args.mode == "render":889+            frame = env.render(mode="rgb_array")890+            if frame is None or frame.ndim != 3 or frame.shape[2] != 3:891+                raise RuntimeError("GGUI renderer did not return an RGB frame")892+            from PIL import Image893+894+            frame_path = output_dir / "render-smoke.png"895+            Image.fromarray(frame).save(str(frame_path))896+            result["frame"] = {897+                "path": str(frame_path),898+                "shape": list(frame.shape),899+                "mean": float(frame.mean()),900+                "std": float(frame.std()),901+            }902+            result["phases"].append("offscreen_render")903+        else:904+            taichi_env = env.taichi_env905+            initial_state = taichi_env.get_state()906+            taichi_env.set_state(initial_state["state"], grad_enabled=True)907+            taichi_env.step(action)908+            loss_info = taichi_env.get_final_loss()909+910+            taichi_env.reset_grad()911+            taichi_env.get_final_loss_grad()912+            taichi_env.step_grad(action)913+            gradient = taichi_env.agent.get_grad(1)914+915+            if not np.all(np.isfinite(gradient)):916+                raise RuntimeError("FluidEngine produced a non-finite action gradient")917+            result["physics"] = {918+                "particles": int(taichi_env.simulator.n_particles),919+                "loss": float(loss_info["loss"]),920+                "gradient_shape": list(gradient.shape),921+                "gradient_l2": float(np.linalg.norm(gradient)),922+                "gradient_nonzero": int(np.count_nonzero(gradient)),923+            }924+            result["phases"].extend(("fluid_forward", "fluid_backward"))925+926+        result["status"] = "passed"927+        result["completed_at"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")928+        atomic_json(result_path, result)929+    except Exception as exc:930+        result["status"] = "failed"931+        result["error"] = "{}: {}".format(type(exc).__name__, exc)932+        result["traceback"] = traceback.format_exc()933+        result["completed_at"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")934+        atomic_json(result_path, result)935+        raise936+937+938+if __name__ == "__main__":939+    main()940diff --git a/cluster/submit_hidden_physics_sweep.py b/cluster/submit_hidden_physics_sweep.py941new file mode 100644942index 0000000..73125fa943--- /dev/null944+++ b/cluster/submit_hidden_physics_sweep.py945@@ -0,0 +1,197 @@946+#!/usr/bin/env python3947+"""Submit or collect isolated FluidLab hidden-physics Slurm lanes."""948+949+import argparse950+import json951+import os952+import re953+import subprocess954+import sys955+import time956+from pathlib import Path957+958+SOURCE_ROOT = Path(__file__).resolve().parents[1]959+sys.path.insert(0, str(SOURCE_ROOT))960+961+from fluidlab.benchmarks.hidden_physics_pouring import (962+    PARAMETER_ANCHORS,963+    parse_values,964+    summarize_parameter,965+)966+967+968+RUN_ROOT = Path("/n/netscratch/ydu_lab/Lab/sjmathy/kempner/fluidlab/runs")969+SUBMIT = Path("/n/home08/sjmathy/.local/bin/fluidlab-submit")970+WORKER = SOURCE_ROOT / "fluidlab/benchmarks/hidden_physics_pouring.py"971+JOB_PATTERN = re.compile(r"Submitted batch job ([0-9]+)")972+973+974+def parse_args():975+    parser = argparse.ArgumentParser(description=__doc__)976+    parser.add_argument("--task", choices=("probe", "precision-fill"), default="probe")977+    parser.add_argument(978+        "--environment",979+        choices=("Pouring-v0", "LeverPouring-v0"),980+        default="Pouring-v0",981+    )982+    parser.add_argument(983+        "--parameter", choices=("mu", "rho", "lam", "all"), default="all"984+    )985+    parser.add_argument("--values", type=parse_values, default=None)986+    parser.add_argument("--time", default="00:15:00")987+    parser.add_argument("--seed", type=int, default=0)988+    parser.add_argument("--steps", type=int, default=None)989+    parser.add_argument("--target-fill", type=float, default=0.25)990+    parser.add_argument("--chunk-steps", type=int, default=25)991+    parser.add_argument("--action-trace", type=Path, default=None)992+    parser.add_argument("--trace-stride", type=int, default=10)993+    parser.add_argument("--sample-particles", type=int, default=256)994+    parser.add_argument("--full-render-trace", action="store_true")995+    parser.add_argument("--manifest", type=Path, required=True)996+    parser.add_argument(997+        "--collect",998+        action="store_true",999+        help="collect existing job results instead of submitting",1000+    )1001+    return parser.parse_args()1002+1003+1004+def atomic_json(path, payload):1005+    path.parent.mkdir(parents=True, exist_ok=True)1006+    temporary = path.with_suffix(path.suffix + ".tmp")1007+    temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")1008+    os.replace(str(temporary), str(path))1009+1010+1011+def selected_parameters(args):1012+    if args.values is not None and args.parameter == "all":1013+        raise ValueError("--values requires one --parameter")1014+    return tuple(PARAMETER_ANCHORS) if args.parameter == "all" else (args.parameter,)1015+1016+1017+def submit(args):1018+    manifest = {1019+        "schema_version": 1,1020+        "task": args.task,1021+        "environment": args.environment,1022+        "robot_driver_trace": (1023+            str(args.action_trace.resolve())1024+            if args.action_trace is not None1025+            else None1026+        ),1027+        "status": "submitted",1028+        "submitted_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),1029+        "jobs": [],1030+    }1031+    atomic_json(args.manifest, manifest)1032+1033+    for parameter in selected_parameters(args):1034+        values = args.values or PARAMETER_ANCHORS[parameter]1035+        for value in values:1036+            command = [1037+                str(SUBMIT),1038+                "--a100",1039+                "--time",1040+                args.time,1041+                str(WORKER),1042+                "--worker",1043+                "--task",1044+                args.task,1045+                "--environment",1046+                args.environment,1047+                "--parameter",1048+                parameter,1049+                "--value",1050+                str(value),1051+                "--seed",1052+                str(args.seed),1053+                "--target-fill",1054+                str(args.target_fill),1055+                "--chunk-steps",1056+                str(args.chunk_steps),1057+                "--trace-stride",1058+                str(args.trace_stride),1059+                "--sample-particles",1060+                str(args.sample_particles),1061+            ]1062+            if args.steps is not None:1063+                command.extend(("--steps", str(args.steps)))1064+            if args.action_trace is not None:1065+                command.extend(("--action-trace", str(args.action_trace.resolve())))1066+            if args.full_render_trace:1067+                command.append("--full-render-trace")1068+            output = subprocess.check_output(command, text=True).strip()1069+            match = JOB_PATTERN.search(output)1070+            if match is None:1071+                raise RuntimeError("could not parse Slurm job ID from: {}".format(output))1072+            job_id = match.group(1)1073+            manifest["jobs"].append({1074+                "parameter": parameter,1075+                "value": float(value),1076+                "job_id": job_id,1077+                "run_dir": str(RUN_ROOT / job_id),1078+                "submit_output": output,1079+            })1080+            atomic_json(args.manifest, manifest)1081+            print("{}={:g}: job {}".format(parameter, value, job_id), flush=True)1082+1083+    return manifest1084+1085+1086+def collect(args):1087+    manifest = json.loads(args.manifest.read_text())1088+    grouped = {}1089+    incomplete = []1090+    failed = []1091+1092+    for job in manifest["jobs"]:1093+        result_path = Path(job["run_dir"]) / "result.json"1094+        if not result_path.exists():1095+            incomplete.append(job["job_id"])1096+            continue1097+        result = json.loads(result_path.read_text())1098+        job["result"] = str(result_path)1099+        job["worker_status"] = result.get("status", "unknown")1100+        if result.get("status") != "passed":1101+            failed.append(job["job_id"])1102+            continue1103+        grouped.setdefault(job["parameter"], []).append(result)1104+1105+    summaries = {}1106+    for parameter, results in grouped.items():1107+        expected = sum(1108+            job["parameter"] == parameter for job in manifest["jobs"]1109+        )1110+        if len(results) == expected:1111+            summaries[parameter] = summarize_parameter(parameter, results)1112+1113+    manifest["parameters"] = summaries1114+    manifest["incomplete_jobs"] = incomplete1115+    manifest["failed_jobs"] = failed1116+    if failed:1117+        manifest["status"] = "failed"1118+    elif incomplete:1119+        manifest["status"] = "running"1120+    else:1121+        manifest["status"] = "passed"1122+        manifest["completed_at"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")1123+    atomic_json(args.manifest, manifest)1124+    print(json.dumps({1125+        "status": manifest["status"],1126+        "incomplete_jobs": incomplete,1127+        "failed_jobs": failed,1128+        "parameters": sorted(summaries),1129+    }, indent=2, sort_keys=True))1130+    return manifest1131+1132+1133+def main():1134+    args = parse_args()1135+    if args.collect:1136+        collect(args)1137+    else:1138+        submit(args)1139+1140+1141+if __name__ == "__main__":1142+    main()1143diff --git a/fluidlab/benchmarks/__init__.py b/fluidlab/benchmarks/__init__.py1144new file mode 1006441145index 0000000..ac5d2281146--- /dev/null1147+++ b/fluidlab/benchmarks/__init__.py1148@@ -0,0 +1,2 @@1149+"""Finite benchmark-development utilities for FluidLab."""1150+1151diff --git a/fluidlab/benchmarks/hidden_physics_pouring.py b/fluidlab/benchmarks/hidden_physics_pouring.py1152new file mode 1006441153index 0000000..3495b9c1154--- /dev/null1155+++ b/fluidlab/benchmarks/hidden_physics_pouring.py1156@@ -0,0 +1,783 @@1157+#!/usr/bin/env python1158+"""Finite hidden-material calibration tasks for FluidLab pouring.1159+1160+This is an engineering Gate 0, not a training-data generator.  It varies one1161+of ``mu``, ``rho``, or ``lam`` while holding the initial particles, appearance,1162+gravity, and (for the probe task) action trace fixed.  Each condition runs in a1163+fresh subprocess because FluidLab's Taichi fields are intentionally large and1164+are most reliably reclaimed when the process exits.1165+1166+Two tasks are available:1167+1168+``probe``1169+    A fixed tilt/hold/return action.  Its action hash must match across hidden1170+    settings.  Use it to determine whether the parameter is observable.1171+1172+``precision-fill``1173+    A privileged, chunked Gate-0 controller that uses collected-particle1174+    fraction as a receiver-fill proxy.  It tests whether feedback can compensate1175+    for the hidden parameter; it is not the final RGB-only adaptive controller.1176+1177+``fixed-angle-fill``1178+    A goal-independent Task-1 rollout driven by a measured Franka action trace.1179+    Every candidate uses the same two-second ramp/hold/return timing; only the1180+    maximum tilt angle changes.  Goal configurations are scored afterwards so1181+    changing 30% to 60% cannot silently change the physical rollout.1182+"""1183+1184+import argparse1185+import hashlib1186+import json1187+import os1188+import subprocess1189+import sys1190+import time1191+from pathlib import Path1192+1193+1194+PARAMETER_ANCHORS = {1195+    "mu": (50.0, 200.0, 800.0),1196+    "rho": (0.5, 1.0, 2.0),1197+    "lam": (69.445, 277.78, 1111.12),1198+}1199+1200+REFERENCE_MATERIAL = {

Showing the first 1,200 of 3559 lines. Download the file for the rest.