CoolFace
Apppublic

tinkersnot/bench-minicpm5-1b

sourceHugging Faceupdated 3mo agoView on Hugging Face
0likes
App README

Nexus Cortex — HF Space model server

Serves an open model for the nexus-cortex harness via the hf-space provider. Exposes a Gradio /run(messages_json, tools_json, max_new_tokens, temperature) -> str API that the harness calls through @gradio/client; raw output is normalized (tool-calls + reasoning) harness-side.

Deploy

  1. 1.hf repo create <you>/<space> --repo-type space --space-sdk gradio (flavor zero-a10g = free GPU on PRO).
  2. 2.hf upload <you>/<space> . --repo-type space (this folder).
  3. 3.Set the model via the MODEL_ID Space variable (default Qwen/Qwen3-0.6B). For reasoning models (Qwen3/SmolLM3), set ENABLE_THINKING=1.

Use in the harness

HF_TOKEN=hf_...            # or HUGGINGFACE_API_KEY / HUGGINGFACE_TOKEN
HF_SPACE_ID=<you>/<space>
HF_SPACE_REASONING=true    # if the model reasons
DEFAULT_MODEL_ID=hf-space  # or HELPER_MODEL_ID=hf-space

Sampling variables (vendor-recommended defaults)

Per-model sampler knobs the OpenAI request shape doesn't carry, set as Space variables: GEN_TOP_P (Qwen/SmolLM3: 0.95), GEN_MIN_P + GEN_REP_PENALTY (LFM2.5: 0.15 / 1.05).

app_vllm.py — paged-KV variant for paid GPUs (multi-turn tool loops)

app.py (transformers) re-prefills the WHOLE conversation every call; the attention spike OOMs 3-4B models at ~16k+ context in multi-turn tool loops. For paid hardware, deploy app_vllm.py as the Space's app.py with requirements.txt = vllm: same /run contract, but prefix caching makes each turn's prefill cover only the new tokens, and the paged KV pool replaces mid-loop CUDA OOMs with a clean MAX_MODEL_LEN ceiling (mirror it in the card's context window).

Hardware constraints (measured, 2026-07-09, vLLM v0.23):

  • —Requires SM80+ (l4x1 / a10g and up). On t4-small (SM75/Turing) the V1 engine's FlashInfer prefill kernel hard-fails (BatchPrefillWithPagedKVCache ... invalid argument) and VLLM_ATTENTION_BACKEND overrides don't switch the prefill path. Keep T4 Spaces on app.py.
  • —NOT ZeroGPU-compatible (needs a persistent CUDA context) — keep app.py there.
  • —Numerics caveat: kernel differences vs transformers can flip near-tie greedy argmax on format-fragile tool callers (measured: Phi-4-mini emits its tool token under transformers but prose under vLLM, both dtypes, identical token ids). Debug with the max_new_tokens=-1 prompt-echo contract.