Modelpublic

Baekpica/Solar-Open2-250B-Mixed-Quant-GGUF

sourceHugging Faceotherupdated 13d agoView on Hugging Face

0likes1kdownloads

Model Card

Solar Open2 250B Mixed-Quant GGUF

Built with Solar.

This repository contains the locked MXQ-v1 candidate derived from `upstage/Solar-Open2-250B`. It preserves the complete topology: 48 layers, all 320 routed experts per layer, one shared expert, and the original [GQA, KDA, KDA, KDA] × 12 schedule. No expert, layer, or parameter pruning, merging, or distillation was performed. This is an independent conversion and is not an official Upstage release.

Verified throughput (2026-09-14, Rust host, 300–2200 MHz cap): On one DGX Spark GB10, SM clock 2190 MHz. MXQ-v1 with K-FP8/V-FP4 KV. Cold ds4-bench, Promessi Sposi, 4,096-token chunks, 64 greedy tokens, MTP off. Interleaved medians of three; 196,608 frontier logits and 64 IDs byte-identical.

Round 1 default-on FATTNWS (`DS4SOLARFATTNWS=0` restores pair):

Prompt tokens	Prefill off → on	Decode off → on
8,192	1,050.86 → 1,075.76 tok/s	17.40 → 17.44
65,536	731.24 → 927.50 tok/s	13.06 → 13.02

Round 2 skip Q3 handoff down sanitize (DS4_CUDA_MOE_HANDOFF_SANITIZE=1 restores the pass), measured with WS on:

Prompt tokens	Prefill off → on	Decode off → on
8,192	1,073.59 → 1,095.61 tok/s	17.43 → 17.43
65,536	925.19 → 943.18 tok/s	13.02 → 13.01

Runtime: `ds4-dfm-rs@67de331`. Disk-KV restart and HTTP partial fork now reuse prefixes. This is not HTTP and not the uncapped September 7/12 benches. Keep the SM cap on GB10. Protocol, serving, rejects.

[image]

The September 7 fused-MoE pass and September 12 uncapped controls use other clocks and are not this graph. 1,048,576-token serving is not claimed.

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.

Artifact

Variant	Split	Size	Approximate BPW
MXQ-v1	11 shards	95,533,532,160 bytes (88.973 GiB)	3.05

Download the complete 11-file split beginning with Solar-Open2-250B-MXQ-v1-00001-of-00011.gguf. Exact hashes are in MXQ-v1-SHA256SUMS.

The full BF16 and full-model Q8_0 reference conversions are published separately in `Baekpica/Solar-Open2-250B-GGUF`.

Precision recipe

Module family	Precision
Token embedding, LM head	Q8_0
GQA q/k/v/o/g projections	Q8_0
KDA matrix projections	Q8_0
Shared expert gate/up/down	Q8_0
Routed gate/up, layers 4–43	IQ2_XXS + imatrix
Routed down, layers 4–43	Q3_K + imatrix
Routed gate/up/down, layers 0–3 and 44–47	Q4_K + imatrix
Router, norms, small KDA control/norm/conv tensors	F32

The final GGUF contains 1,083 tensors: 409 F32, 530 Q80, 80 IQ2XXS, 40 Q3K, and 24 Q4K tensors. quant-recipe-v1.yaml and verify-v1.json contain the machine-readable recipe and verification result.

Calibration

The Solar chat-template calibration corpus follows the published mix from Baekpica/Solar-Open2-120B-A15B-REAM-148E-Healing-Mix: instruction chat 22%, Cascade stage-1 reasoning 16%, Cascade stage-2 reasoning 16%, Korean 16%, other multilingual 12%, finance 6%, SWE 6%, and algorithmic code 6%.

The imatrix processed 590 × 512-token chunks. Layers 1–47 reached strict 320/320 routed-expert coverage. Layer 0 retained one explicit, pinned exception of 29 zero-activation experts; the exact IDs and gate are recorded in verify-imatrix.json. No precision was lowered to compensate for this exception.

Validation status

—All 11 shards passed split ordering, bounds, topology, tensor count, tensor family, and exact tensor-type verification with zero errors.
—The artifact loaded resident across four H100 NVL GPUs and completed 32 deterministic short generations plus eight longer no-think quality checks.
—Every candidate output was non-empty with valid token IDs. Fatal/content checks passed for Korean, English, Japanese, code, arithmetic, exact JSON, and constrained-list prompts; no NaN or malformed structured output was observed.
—A development ds4 CUDA runtime mapped the complete artifact on one H100 and passed full-weight, recurrent-state snapshot/restore, replay, and cold invalidation regressions.
—A native sm_121a ds4 build serves the complete artifact on one DGX Spark GB10 through the OpenAI-compatible server at -c 196608 with three persistent banks. See DGX Spark for the measured point.

H100 timings are development and artifact-sanity evidence only. Do not present them as DGX Spark performance. 1,048,576-token serving on Spark is not claimed.

DGX Spark

One NVIDIA DGX Spark GB10 (128 GB unified memory, driver 610.43.02, CUDA 13.3, sm_121a). Runtime `Baekpica/ds4@b2e52b9` on the dfm line. Artifact: this MXQ-v1 11-shard split. A long-lived VMM weight owner (--backend vmm --scope base --reserve-gb 16, 453 derived aligned artifacts) plus a restartable ds4-server worker:

DS4_CUDA_WEIGHT_IPC_MANIFEST="$RUN/weights.manifest" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
DS4_SERVER_COALESCE_MAX=3 \
DS4_SERVER_COALESCE_MAX_TOKENS=4096 \
DS4_CONT_PREFILL_CHUNK=4096 \
DS4_METAL_PREFILL_CHUNK=4096 \
./ds4-server -m Solar-Open2-250B-MXQ-v1-00001-of-00011.gguf \
  --cuda -c 196608 -n 1024 -t 6 \
  --host 0.0.0.0 --port 8002 --model-id solar-open2-250b \
  --no-spec --no-update-check --mem-floor-gb 8

OpenAI Chat Completions, thinking disabled, exact-cold (cached_tokens=0), 128 decode tokens, median of three. Loaded SM clocks 2,411–2,561 MHz (uncapped; not the 300–2200 MHz campaign above). banks_total=3 still admitted.

Depth	Prompt tokens	Prefill	Decode p50	Decode API
8K	8,222	1,050.7 tok/s	19.05 tok/s	18.9 tok/s
64K	66,761	804.5 tok/s	13.07 tok/s	14.1 tok/s

On the same host and artifact, before the dfm default-path series that ends at b2e52b9, 8K decode was 17.5 tok/s and 64K average prefill was 710 tok/s. The series kept the production path as: skip width-1 inner CUDA graph capture; walk FATTN KV in 32-key then GQA-pair 64-key tiles; keep the KDA 64 KiB state tile in shared memory; tile grouped GQA decode KV in 32 keys with packed K-FP8/V-FP4 dequant.

test-solar-kv on this binary: 512-token GQA2 vs one-head rel_rms=0; split vs direct rel_rms=8.45e-7. Incremental T(64K)−T(60K) last-4K is not a published figure. Native 1,048,576-token metadata is not a measured Spark pass.

Partial prefix reuse

Runtime `Baekpica/ds4@042bcea` adds a bounded, shared Solar KDA checkpoint pool to the continuous lane. The 32 slots are demand-mapped, exact forks share references, and long prefills retain roughly 24 periodic checkpoints across the configured context in addition to request boundaries. A divergent prompt restores the nearest safe checkpoint and replays only the gap.

One controlled A/B run used the same GB10, artifact, four-bank worker, -c 65536, 4,096-token prefill chunks, and greedy no-think Chat requests. The control set DS4_SERVER_FORK_PARTIAL=0, which also disables checkpoint reservation. The treatment used the default.

Request	Prompt	Treatment cached	Control TTFT	Treatment TTFT	Speedup
Source	12,123	0	10,654.0 ms	10,710.7 ms	0.995×
Branch at ~6K	6,079	4,096	5,323.7 ms	1,868.8 ms	2.85×
Branch at ~10K	10,148	8,192	8,920.2 ms	1,929.8 ms	4.62×

Both branch responses were byte-identical between control and treatment. Treatment reported two partial-fork admits, 12,288 cached tokens, zero failed requests, and zero admission rejects. Source checkpoint capture added 56.7 ms (0.53%) in this single run. After the three requests, device-live memory was 102.04 GiB and system-available memory was 14.59 GiB with an 8 GiB floor.

This is a bounded worker-local cache, not an unbounded radix tree. Token history and GQA prefix rows remain anchored to a retained bank, and the KDA checkpoint pool is not serialized into disk-KV payloads.

On 2026-09-14, a one-bank worker at -c 8192 with --kv-disk-dir and partial fork, under the 300–2200 MHz cap, reused cached_tokens=538 after a clean restart continuation and cached_tokens=4096 on a 4,177-token divergent Chat (81 computed, TTFT 416 ms vs 4.7 s). Prompts shorter than the 4,096-token stride still have no interior checkpoint.

Provenance

—Source: upstage/Solar-Open2-250B@9190fbe63a2ad8e17fc766ccceb36de7c66f004b
—Official implementation oracle: upstageAI/transformers@ef5c37485091057070b36dd0620798f74988eb01
—llama.cpp conversion base: ggml-org/llama.cpp@6ea215d171fd31df943bf1ac8227129f2b963160
—Solar conversion/runtime patch SHA-256: 7c267850de7a08af66a9bd64e9d4a3b33e085a4d2888d94a4c3d919f87283e44
—Reproduction repository: `Baekpica/solar-open2-mixed-ds4`

Current upstream runtimes may not yet recognize the solar-open2 GGUF architecture. Use a compatible Solar Open 2 llama.cpp port or the linked ds4 development branch; compatibility claims should be tied to an exact runtime revision.

License and attribution

The source model and this derivative artifact are distributed under the included Upstage Solar License. The model name begins with Solar, and this card prominently states “Built with Solar” as required by that license.