CoolFace
Modelpublic

soyelmismo/laya-multilingual-onnx

sourceHugging Faceapache-2.0updated 5d agoView on Hugging Face
2likes
Model Card

Laya Multilingual — CPU-Optimized ONNX Checkpoints

This repository provides high-speed, CPU-optimized ONNX checkpoints for convaiinnovations/laya-multilingual (322M parameters, mmBERT architecture).

These weights are engineered specifically for real-time prompt classification, semantic decision routing, and System One fast-path inference with 0.0 ms network latency on commodity CPUs.

Why CPU-Optimized Checkpoints?

The community WebGPU export (mizchi/laya-multilingual-onnx) is formatted in FP16. On standard CPUs without native half-precision vector execution units (such as x86_64 or ARM Neoverse N1/Cortex-A76), ONNX Runtime emulates FP16 in software, resulting in ~3,900 ms latency per sequence.

This repository provides two calibrated alternatives:

Checkpoint FilePrecisionRAM / DiskCPU LatencyOptimization & Target Hardware
`model.onnx` (Default) / `model-int8.onnx`QInt8325 MB250 – 300 msDynamic per-channel quantization. Accelerated via ARMv8.2-A asimddp instructions (sdot/udot) and x86 AVX-512 VNNI. Sensitive layers (scorer, act_head, type_emb) remain in full precision to preserve 100% classification accuracy.
`model-fp32.onnx`Float321.29 GB550 – 650 msFull precision FP32 weights. 100% bit-exact parity with PyTorch. Optimized for ARM NEON and x86 AVX2.
Original WebGPU exportFloat16617 MB~3,900 msEmulated via software on CPUs lacking native FP16 instructions.

Files in this Repository

  • —`model.onnx`: The selective INT8 model (325 MB). Default for instant, low-latency CPU routing.
  • —`model-int8.onnx`: Explicit alias of the INT8 model.
  • —`model-fp32.onnx`: The full-precision FP32 model (1.29 GB).
  • —`tokenizer.json` & `tokenizer/`: Fast tokenizer files for Hugging Face tokenizers.
  • —`rl_agent_config.json`: Calibrated temperature scaling and category thresholds.

Quickstart with OpenProxy

OpenProxy automatically detects these models for native, in-process combo decision routing.

1. Download into the default OpenProxy directory

bash
mkdir -p ~/.openproxy/models/laya
huggingface-cli download soyelmismo/laya-multilingual-onnx \
  --local-dir ~/.openproxy/models/laya \
  --include "model.onnx" "tokenizer.json" "rl_agent_config.json"

Or via direct HTTP curl:

bash
mkdir -p ~/.openproxy/models/laya
HF_BASE="https://huggingface.co/soyelmismo/laya-multilingual-onnx/resolve/main"

curl -L -o ~/.openproxy/models/laya/model.onnx "$HF_BASE/model.onnx"
curl -L -o ~/.openproxy/models/laya/tokenizer.json "$HF_BASE/tokenizer.json"
curl -L -o ~/.openproxy/models/laya/rl_agent_config.json "$HF_BASE/rl_agent_config.json"

2. Run in Docker

Mount the models folder into the container:

yaml
services:
  openproxy:
    image: ghcr.io/soyelmismo/openproxy:latest
    ports:
      - "8787:8787"
    volumes:
      - ./config.toml:/etc/openproxy/config.toml:ro
      - ~/.openproxy/models/laya:/var/lib/openproxy/models/laya:ro

OpenProxy scans /var/lib/openproxy/models/laya and initializes native in-process inference without any Python dependencies or daemons.

License & Attribution