Matrix-Corp/Matrix-Voxel

Matrix Voxel — Full Architecture & Planning Document

3D Generation Model Family | Matrix.Corp

Family Overview

Matrix Voxel is Matrix.Corp's 3D generation family. Five models sharing a common flow-matching backbone, each with task-specific decoder heads. Four specialist models are open source; one unified all-in-one (Voxel Prime) is closed source and API-only.

Input Modalities (All Models)

Every Voxel model accepts any combination of:

All inputs projected to a shared 1024-dim conditioning embedding space before entering the backbone.

Core Architecture — Shared Flow Matching Backbone

Why Flow Matching?

Flow matching (Lipman et al. 2022, extended by Stable Diffusion 3 / FLUX lineage) learns a direct vector field from noise → data. Faster than DDPM diffusion (fewer inference steps, typically 20–50 vs 1000), more stable training, better mode coverage. State of the art for generative models as of 2025–2026.

3D Representation — Triplane + Latent Voxel Grid

All Voxel models operate in a shared latent 3D space:

• —Triplane representation: three axis-aligned feature planes (XY, XZ, YZ), each 256×256×32 channels
• —Any 3D point queried by projecting onto all 3 planes and summing features
• —Compact (3 × 256 × 256 × 32 = ~6M latent values) yet expressive
• —Flow matching operates on this triplane latent space, not raw 3D points
• —Decoder heads decode triplane to task-specific output format

Backbone Architecture

VoxelBackbone
├── Input Encoder (multimodal conditioning)
│   ├── TextEncoder         — T5-XXL + CLIP-ViT-L, projected to 1024-dim
│   ├── ImageEncoder        — DINOv2-L, projected to 1024-dim  
│   ├── MultiViewEncoder    — custom transformer over N views
│   ├── VideoEncoder        — Video-MAE, temporal pooling → 1024-dim
│   └── PointCloudEncoder   — PointNet++, global + local features → 1024-dim
│
├── Conditioning Fusion
│   └── CrossModalAttention — fuses all active input modalities
│
├── Flow Matching Transformer (DiT-style)
│   ├── 24 transformer blocks
│   ├── Hidden dim: 1536
│   ├── Heads: 24
│   ├── Conditioning: AdaLN-Zero (timestep + conditioning signal)
│   ├── 3D RoPE positional encoding for triplane tokens
│   └── ~2.3B parameters
│
└── Triplane Decoder (shared across all specialist models)
    └── Outputs: triplane feature tensor (3 × 256 × 256 × 32)

Flow Matching Training

• —Learn vector field vθ(xt, t, c) where x_t is noisy triplane, c is conditioning
• —Optimal transport flow: straight paths from noise → data (better than DDPM curved paths)
• —Inference: 20–50 NFE (neural function evaluations) — fast on A100
• —Classifier-free guidance: unconditional dropout 10% during training
• —Guidance scale 5.0–10.0 at inference

Task-Specific Decoder Heads

Each specialist model adds a decoder head on top of the shared triplane output.

Voxel Atlas — World Generation Decoder

Task: Generate full 3D environments and worlds — terrain, buildings, vegetation, interior spaces.

Output formats:

• —Voxel grids (.vox, Magica Voxel format) — for Minecraft-style worlds
• —OBJ scene (multiple meshes with materials) — for Unity/Unreal environments
• —USD stage (.usd) — industry standard scene format

Decoder head:

TriplaneAtlasDecoder
├── Scene Layout Transformer
│   ├── Divides space into semantic regions (terrain, structures, vegetation, sky)
│   └── 6-layer transformer over 32×32 spatial grid of scene tokens
├── Region-wise NeRF decoder (per semantic region)
│   └── MLP: 3D coords + triplane features → density + RGB + semantic label
├── Marching Cubes extractor → raw mesh per region
├── Scene graph assembler → parent-child relationships between objects
├── Voxelizer (for .vox output) → discretizes to user-specified resolution
└── USD exporter → full scene hierarchy with lighting + materials

Special modules:

• —Infinite world tiling: generate seamless adjacent chunks that stitch together
• —Biome-aware generation: desert, forest, urban, underwater, space, fantasy
• —LOD generator: auto-generates 4 levels of detail per scene object
• —Lighting estimator: infers plausible sun/sky lighting from scene content

Typical generation sizes:

• —Small scene: 64×64×64 voxels or ~500m² OBJ scene — ~8 seconds on A100
• —Large world chunk: 256×256×128 voxels — ~35 seconds on A100

Voxel Forge — Mesh / Asset Generation Decoder

Task: Generate clean, game-ready 3D assets — characters, objects, props, vehicles, architecture.

Output formats:

• —OBJ + MTL (universal)
• —GLB/GLTF (web & real-time)
• —FBX (game engine standard)
• —USDZ (Apple AR)

Decoder head:

TriplaneForgeDec oder
├── Occupancy Network decoder
│   └── MLP: 3D point + triplane → occupancy probability
├── Differentiable Marching Cubes → initial raw mesh
├── Mesh Refinement Network
│   ├── Graph neural network over mesh vertices/edges
│   ├── 8 message-passing rounds
│   └── Predicts vertex position offsets → clean topology
├── UV Unwrapper (learned, SeamlessUV lineage)
├── Texture Diffusion Head
│   ├── 2D flow matching in UV space
│   ├── Albedo + roughness + metallic + normal maps
│   └── 1024×1024 or 2048×2048 texture atlas
└── LOD Generator → 4 polycount levels (100% / 50% / 25% / 10%)

Special modules:

• —Topology optimizer: enforces quad-dominant topology for animation rigs
• —Symmetry enforcer: optional bilateral symmetry for characters/vehicles
• —Scale normalizer: outputs at real-world scale (meters) with unit metadata
• —Material classifier: auto-tags materials (metal, wood, fabric, glass, etc.)
• —Animation-ready flag: detects and preserves edge loops needed for rigging

Polygon counts:

• —Low-poly asset: 500–5K triangles — ~6 seconds on A100
• —Mid-poly asset: 5K–50K triangles — ~15 seconds on A100
• —High-poly asset: 50K–500K triangles — ~45 seconds on A100

Voxel Cast — 3D Printable Generation Decoder

Task: Generate physically valid, printable 3D models. Watertight, manifold, structurally sound.

Output formats:

• —STL (universal printing format)
• —OBJ (watertight)
• —STEP (CAD-compatible, parametric)
• —3MF (modern printing format with material data)

Decoder head:

TriplaneCastDecoder
├── SDF (Signed Distance Field) decoder
│   └── MLP: 3D point + triplane → signed distance value
├── SDF → Watertight Mesh (dual marching cubes, no holes guaranteed)
├── Printability Validator
│   ├── Wall thickness checker (min 1.2mm enforced)
│   ├── Overhang analyzer (>45° flagged + support detection)
│   ├── Manifold checker + auto-repair
│   └── Volume/surface area calculator
├── Support Structure Generator (optional)
│   └── Generates minimal support trees for FDM printing
├── STEP Converter (via Open CASCADE bindings)
└── Slicer Preview Renderer (preview only, not full slicer)

Special modules:

• —Structural stress analyzer: basic FEA simulation to detect weak points
• —Hollowing engine: auto-hollows solid objects with configurable wall thickness + drain holes
• —Interlocking part splitter: splits large objects into printable parts with snap-fit joints
• —Material suggester: recommends PLA / PETG / resin based on geometry complexity
• —Scale validator: ensures object is printable at specified scale on common bed sizes (Bambu, Prusa, Ender)

Validation requirements (all Cast outputs must pass):

• —Zero non-manifold edges
• —Zero self-intersections
• —Minimum wall thickness ≥ 1.2mm at requested scale
• —Watertight (no open boundaries)

Voxel Lens — NeRF / Gaussian Splatting Decoder

Task: Generate photorealistic 3D scenes represented as Neural Radiance Fields or 3D Gaussian Splats — primarily for visualization, VR/AR, and cinematic rendering.

Output formats:

• —.ply (3D Gaussian Splatting — compatible with standard 3DGS viewers)
• —NeRF weights (Instant-NGP / Nerfstudio compatible)
• —MP4 render (pre-rendered orbital video)
• —Depth maps + normal maps (per-view, for downstream use)

Decoder head:

TriplaneLensDecoder
├── Gaussian Parameter Decoder
│   ├── Samples 3D Gaussian centers from triplane density
│   ├── Per-Gaussian: position (3), rotation (4 quaternion), scale (3),
│   │   opacity (1), spherical harmonics coefficients (48) → color
│   └── Targets: 500K–3M Gaussians per scene
├── Gaussian Densification Module
│   ├── Adaptive densification: split/clone in high-gradient regions
│   └── Pruning: remove low-opacity Gaussians
├── NeRF branch (parallel)
│   ├── Hash-grid encoder (Instant-NGP style)
│   └── Tiny MLP: encoded position → density + color
├── Rasterizer (differentiable 3DGS rasterizer)
│   └── Used during training for photometric loss
└── Novel View Synthesizer
    └── Renders arbitrary camera trajectories for MP4 export

Special modules:

• —Lighting decomposition: separates scene into albedo + illumination components
• —Dynamic scene support: temporal Gaussian sequences for animated scenes (from video input)
• —Background/foreground separator: isolates subject from environment
• —Camera trajectory planner: auto-generates cinematic orbital/fly-through paths
• —Compression module: reduces 3DGS file size by 60–80% with minimal quality loss

Generation modes:

• —Object-centric: single object, orbital views — ~12 seconds on A100
• —Indoor scene: full room with lighting — ~40 seconds on A100
• —Outdoor scene: landscape or street — ~90 seconds on A100

Voxel Prime — Closed Source All-in-One

Access: API only. Not open source. Weights never distributed.

Voxel Prime contains all four decoder heads simultaneously, plus:

Additional Prime-only modules:

• —Cross-task consistency: ensures Atlas world + Forge assets + Lens scene all match when generated together
• —Scene population engine: generates a world (Atlas) then auto-populates it with assets (Forge)
• —Pipeline orchestrator: chains Atlas → Forge → Cast → Lens in one API call
• —Photorealistic texture upscaler: 4× super-resolution on all generated textures
• —Style transfer module: apply artistic style (e.g. "Studio Ghibli", "cyberpunk", "brutalist architecture") across all output types
• —Iterative refinement: text-guided editing of already-generated 3D content

API endpoint:

POST /v1/voxel/generate
{
  "prompt": "A medieval castle on a cliff at sunset",
  "output_types": ["world", "mesh", "nerf"],  # any combination
  "inputs": {
    "image": "base64...",       # optional reference image
    "multiview": ["base64..."], # optional multi-view images
    "video": "base64...",       # optional video
    "model": "base64..."        # optional existing 3D model
  },
  "settings": {
    "quality": "high",          # draft | standard | high
    "style": "realistic",       # realistic | stylized | low-poly | ...
    "scale_meters": 100.0,      # real-world scale
    "symmetry": false,
    "printable": false
  }
}

Shared Custom Modules (All Models)

Training Data Plan

Parameter Estimates

All fit on A100 40GB in BF16. INT8 quantization brings all under 15GB (consumer 4090 viable).

Training Strategy

Phase 1 — Backbone Pre-training

• —Train shared backbone on Objaverse-XL triplane reconstructions
• —Learn general 3D structure without task-specific heads
• —Context: text + single image conditioning only
• —100K steps, A100 cluster

Phase 2 — Decoder Head Training (parallel)

• —Freeze backbone, train each decoder head independently
• —Atlas: ScanNet + synthetic world data
• —Forge: ShapeNet + Objaverse + texture data
• —Cast: Thingiverse + watertight synthetic meshes
• —Lens: Polycam + synthetic multi-view renders
• —50K steps each

Phase 3 — Joint Fine-tuning

• —Unfreeze backbone, fine-tune end-to-end per specialist model
• —Add all input modalities (video, multi-view, point cloud)
• —30K steps each

Phase 4 — Prime Training

• —Initialize from jointly fine-tuned backbone
• —Train all decoder heads simultaneously
• —Cross-task consistency losses
• —Prime-only module training (pipeline orchestrator, style transfer)
• —50K steps

HuggingFace Plan

Matrix-Corp/Voxel-Atlas-V1    — open source
Matrix-Corp/Voxel-Forge-V1    — open source
Matrix-Corp/Voxel-Cast-V1     — open source
Matrix-Corp/Voxel-Lens-V1     — open source
Matrix-Corp/Voxel-Prime-V1    — closed source, API only (card only, no weights)

Collection: Matrix-Corp/voxel-v1

Status

• —🔴 Planned — Architecture specification complete
• —Backbone design finalized
• —Decoder head designs finalized
• —Training data sourcing: TBD
• —Compute requirements: significant (A100 cluster for training)
• —Timeline: TBD