aksern/rgsnr-v3-unet-0.1

RGSNR — Renderer-Guided Sparse Neural Rendering

Experimental checkpoint for sparse tile inpainting in game frames. This model is under testing and not production-ready. Quality is limited — see outputs below and limitations section.

• —Model: rgsnr_unet (4.03M params, base_channels 32, partial convolutions, early temporal fusion)
• —Input: 11 channels — sparse RGB (3) + mask (1) + Sobel depth (1) + 2 temporal frames (6) — at 360x640
• —Output: 3-channel RGB reconstruction at 360x640
• —Tile size: 32x32, binary mask

Repository: https://github.com/aksern-ai/RGSNR/ Config: config.json — full resolved config including inference and training metadata.

Status

• —Version: CNN v3, second training run
• —Epochs: 6, best epoch 4, early stopped / undertrained
• —Validation PSNR 23.00 dB / SSIM 0.786 / hole PSNR 19.63 dB — moderate, with visible artifacts in holes
• —Trained on gameplay images from Kaggle: https://www.kaggle.com/datasets/aditmagotra/gameplay-images (10 games, 8000 train / 1000 val / 1000 test, 80/10/10 stratified)
• —Intended for research and testing only

Outputs

Each image below is a vertical panel (644x1088) with three rows:

Top: ground truth — Middle: sparse input (black tiles were not rendered, 50% of tiles kept) — Bottom: model reconstruction

Both use the same hold-out frame at 50% rendered ratio, with different mask strategies. No cherry-picking; these are the two files in test_images/.

Notes:

• —Checkerboard uses a regular alternating pattern. Random tile uses stochastic tile selection.
• —Middle row shows exactly which tiles the model received. The model must fill the black regions using depth, mask, and temporal context.
• —Bottom row shows typical artifacts: blur in high-frequency areas, color bleeding across depth edges, loss of thin geometry. Quality drops further at lower ratios (e.g., 0.3) and improves at higher ratios (e.g., 0.7), but does not reach ground truth.
• —Temporal input is duplicated frames with fake affine motion during training, not real motion vectors.

Training details

• —Config: configs/default.yaml, device cuda, 339s/epoch
• —Image size 360x640, depth_mode sobel, temporal true (2 frames, fake motion)
• —Mask: tile 32, train ratios [0.3, 0.5, 0.7] sampled per batch, strategies [randomtile, checkerboard, contentaware] (threshold 0.35), binary
• —Loss: L1 (hole_weight 3.0) + SSIM 0.2 + perceptual 0.01
• —Optimizer: AdamW lr 1.5e-4, weightdecay 1e-4, cosine scheduler, mixed precision, gradclip 1.0, batch 8
• —Epoch 005/6: loss 0.1617, val PSNR 22.99 SSIM 0.7864 hole 19.63, best 23.00 (e4), lr 1.00e-05

See config.json for full values including inference and meta (git commit 9ab4858, created 2026-09-18).

Evaluation

Validation metrics are from in-distribution split only. No leave-one-game-out test was run for this checkpoint. To reproduce a ratio sweep:

python eval.py --ckpt checkpoints/model.pt --ratios 0.1 0.3 0.5 0.7 0.9 1.0 --save_dir demo_output

Usage

Requires the repository code. Checkpoint auto-discovers config.json next to model.pt.

Single image:

python demo.py --ckpt checkpoints/model.pt --img data/Minecraft/image_0.png --ratio 0.5 --strategy random_tile --out demo_output/demo.png

Batch / directory:

python demo.py --ckpt checkpoints/model.pt --img "data/Forza Horizon/" --ratio 0.5 --strategy checkerboard --out demo_output/ --limit 8

Outputs include *_mask.png and *_depth.png sidecars plus a .json with parameters.

Python API:

import torch
from rgsnr.models import RGSNRUNet
from rgsnr.data.dataset import load_rgb, synthetic_depth_sobel
from rgsnr.data.masks import MaskGenerator

device = "cuda" if torch.cuda.is_available() else "cpu"
ckpt = torch.load("checkpoints/model.pt", map_location=device)
model = RGSNRUNet(in_channels=11, base_channels=32, temporal_frames=2, use_partial_conv=True).to(device)
model.load_state_dict(ckpt["model"])
model.eval()

H, W = 360, 640
gt = load_rgb("data/Apex Legends/image_0.png", (H, W)).unsqueeze(0).to(device)
gray = 0.2989*gt[:,0:1] + 0.5870*gt[:,1:2] + 0.1140*gt[:,2:3]
mg = MaskGenerator(tile_size=32)
mask = mg(H, W, gray=gray[0], ratio=0.5, strategy="random_tile", device=device).unsqueeze(0)
depth = synthetic_depth_sobel(gt[0]).unsqueeze(0)
temporal = torch.stack([gt[0]]*2, dim=0).unsqueeze(0)
sparse = gt * mask
with torch.no_grad():
    reconstructed, _ = model(sparse, mask, depth, temporal)

Limitations

• —Undertrained (6 epochs). Expect blur, temporal instability, and failure on disocclusion, particles, transparency, foliage, and reflections.
• —Depth is a synthetic Sobel proxy, not a true depth buffer. Motion is simulated, not real vectors.
• —Requires engine integration (mask + depth + temporal). Not a drop-in post-process filter.
• —No perceptual/LPIPS/GAN tuning in this run (weights 0). Results are optimized for L1/SSIM.
• —Single-resolution (360x640). No high-resolution or variable-rate shading hardware path tested.
• —Evaluation is limited to the same dataset distribution. Generalization to other games or styles is untested.

License

OpenMDW-1.1