Modelpublic

fr0stbit3/laya-gguf

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

4likes1.2kdownloads

Model Card

GGUF conversion of convaiinnovations/laya (Apache-2.0), f16, via llama.cpp convert_hf_to_gguf.py. laya-F16.gguf (and the quants) hold the ModernBERT backbone only, and load in llama.cpp as modern-bert (use --embeddings --pooling none). The decision head and config are in the sibling file laya-head.safetensors (like an mmproj file); read it with laya_head.py (load_head(path) returns config + head weights under the original PyTorch names). The head itself must run outside llama.cpp. Lightly tested: outputs matched the original on a limited set (100 AG News + 100 DAIR Emotion samples, a few examples); may need further testing. Quantized files (laya-Q8_0.gguf, -Q6_K.gguf, -Q4_K_M.gguf; laya-F16.gguf is the unquantized 16-bit original conversion) are made with llama-quantize from the f16 file and are even less tested: checked on a single example only, where Q80/Q6K stayed close to f16 and Q4KM drifted slightly more (probabilities shifted by up to about 0.02 to 0.03, and score outputs by about 0.04). Prefer f16 or Q80 for anything important, and verify Q4K_M on your own data.

Quant check (single example: the HF README ticket "Duplicate charge on invoice 4411"; department=billing probability, urgency score 0-2, churn_risk noul; head weights from the head file, backbone in llama.cpp). One example only, not a benchmark.

file	size	billing p	urgency	churn
HF original (PyTorch)	-	0.967	1.630	0.198
F16	933M	0.966	1.631	0.201
Q8_0	563M	0.967	1.637	0.198
Q6_K	485M	0.970	1.650	0.198
Q4KM	414M	0.967	1.601	0.219

llama.cpp quickstart

bash

# 1. serve the backbone (per-token hidden states; -ub must cover your longest input)
llama-server -m laya-F16.gguf --embeddings --pooling none -c 2048 -ub 2048 -b 2048 --port 8080

# 2. python deps for the decision head + tokenizer
pip install laya requests torch safetensors

python

# quickstart.py: llama.cpp backbone + Laya head from the sibling head file
import types, requests, torch, laya
from laya_head import load_head

HEAD = "laya-head.safetensors"
agent = laya.load("convaiinnovations/laya", device="cpu")  # builds the head architecture + tokenizer/prompt logic

# swap in the decision-head weights from the head file
_, head = load_head(HEAD)
sd = agent.model.state_dict()
for k, v in head.items():
    if k in sd: sd[k].copy_(v)

# swap the PyTorch encoder for the llama.cpp server
D = agent.model.encoder.config.hidden_size
def llamacpp_encoder(input_ids, attention_mask=None, **_):
    out = []
    for i, row in enumerate(input_ids):
        n = int(attention_mask[i].sum())
        r = requests.post("http://localhost:8080/embedding", json={"content": [row[:n].tolist()]}).json()
        h = torch.zeros(input_ids.shape[1], D); h[:n] = torch.tensor(r[0]["embedding"]); out.append(h)
    return types.SimpleNamespace(last_hidden_state=torch.stack(out))
agent.model.encoder.forward = llamacpp_encoder

result = agent.predict(
    {"subject": "Duplicate charge on invoice 4411",
     "body": "We were billed twice for March. Please refund the duplicate."},
    {"department": {"type": "choice", "instructions": "Which team should handle this?",
                    "criteria": {"billing": "invoices, payments, refunds",
                                 "technical": "bugs and outages", "sales": "pricing"}}},
)
print(result["answers"]["department"]["choice"])  # billing

Notes: the laya package still downloads the original weights once (for the head architecture and tokenizer); inference runs the backbone in llama.cpp. Tokenize with the HF tokenizer and send token ids ("content": [ids]) as above. A native Go/C++ head is not provided.

Original model

Full model card, usage, benchmarks and license terms: [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya).