Kentucky-Open-Science/KOS-V5-Instruct

1---2license: apache-2.03library_name: transformers4pipeline_tag: text-generation5language:6- en7developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation)8affiliations:9- name: University of Kentucky, College of Medicine Office for Research10  url: https://medicine.uky.edu/sites/research11- name: University of Kentucky, Center for Clinical and Translational Science (CCTS)12  url: https://www.ccts.uky.edu/13- name: University of Louisville, Kentucky Center for Digital Innovation14  url: https://centers.louisville.edu/kentucky-center-digital-innovation15model_name: KOS-V5-Instruct16model_codename: Catbird17model_type: qwen318base_model: Kentucky-Open-Science/KOS-V5-Base19tags:20- medical21- clinical22- biomedical23- instruction-following24- tool-calling25- function-calling26- from-scratch27- qwen328- KOS-V529- university-of-kentucky30- university-of-louisville31- ccts32---33 34<p align="center">35  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>36</p>37 38# KOS-V5-Instruct · *"Catbird"*39 40**Developed by**41 42**University of Kentucky**43- [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)44- [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)45 46**University of Louisville**47- [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)48 49**A 3.72B-parameter medical language model trained from scratch.** It is not distilled, not pruned and not50continued-pretrained from a general base. KOS-V5 (codename **Catbird**) is the fifth-generation Kentucky Open51Science model line. This repository holds the **instruction-tuned head** of that line: the52[KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) pretraining checkpoint, taken through53SFT and two GRPO reinforcement-learning legs.54 55Unlike the base, this model **follows instructions and calls tools**. It is the downstream SFT/RL artifact that56[KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) was built to initialise.57 58**Code name: Catbird.** The KOS-V5 series is nicknamed Catbird; native to Kentucky, the Gray Catbird59(*Dumetella carolinensis*) is a medium-sized songbird famous for its distinct, cat-like "meow" call. This LLM60was trained completely from scratch by teams from the University of Kentucky (Cat) and University of Louisville61(Bird), so the code name is fitting.62 63> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any64> commercial, clinical, legal, or production-grade application. The user assumes all risks associated with its use.65 66---67 68Its instruction ability comes from **GRPO reinforcement learning** against the *official* IFEval verifier, and its69tool-calling ability from a **second GRPO leg** against the *official* BFCL AST checker, on a base that ranks70**first of 17** at modelling held-out clinical text.71 72**IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2` — the exact metric the73Hugging Face Open LLM Leaderboard publishes as "IFEval."74 75| IFEval **strict-avg** | model | who built it, and how |76| --: | :-- | :-- |77| **72.19** | **KOS-V5-Instruct (ours)** | University research team, 235B tokens, from scratch |78| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |79| **61.6** | [KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct) (previous generation) | University research team, 180B tokens, 24 GPUs |80| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |81 82KOS-V5-Instruct **improves on KOS-V4-Instruct across every benchmark measured**: IFEval strict-avg83**61.6 → 72.19** (+10.6), MMLU **0.2782 → 0.4512** (+17.3), medical QA (**PubMedQA 0.7060**, MedQA 0.3802,84MedMCQA 0.3648 — all up on V4), and official BFCL function-calling85**72.75/73.00/60.50 → 85.00/84.00/80.50** (+12.3 / +11.0 / +20.0). It clears the original GPT-3.5-turbo86generation and the commercially trained Qwen2.5-3B on instruction following, and its tool calling now runs87**above the Qwen3-4B-Instruct-2507 peer**.88 89## Core specifications90 91| Attribute | Detail |92| :--- | :--- |93| **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |94| **Parameters** | 3.715 B |95| **Hidden / Layers** | 2560 / 36 |96| **Attention** | 32 query / 8 KV heads (GQA 4:1), head_dim 128, per-head QK-RMSNorm |97| **Feed-forward** | SwiGLU, intermediate 9728 |98| **Vocabulary** | 32,000, custom medical byte-level BPE |99| **Context length** | 32,768 |100| **Position encoding** | RoPE, θ = 25,000 |101| **Embeddings** | tied |102| **Precision** | bfloat16 (7.43 GB, single shard) |103 104## Pre-training (the KOS-V5 base)105 106Fine-tuned from [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) — the from-scratch107pretrained foundation, a complete single-epoch run over **235.2B tokens**. See that card for corpus composition and108disclosed pretraining issues.109 110## Post-training (this model)111 112Three stages on top of the base — no LoRA, no distillation, no reward model, no LLM judge.113 114**SFT** — one shuffled full-parameter pass over a **736,990-record / 1.32B-token** audited instruction mix115(ChatML). The mix was deduplicated, instruction-collision canonicalized, structurally validated, per-record116decontaminated and BFCL-closure scanned; clinical FHIR records were dropped and tool-record system prompts117de-welded so tool use binds to the request rather than to a fixed frame.118 119**RL leg 1 — instruction following (GRPO via verl)** — deterministic verifiable reward. The reward is the120**official** `lm_eval` IFEval instruction registry — the same checkers the benchmark scores with, not a121re-implementation. One 8-GPU wall, 164 steps, KL 0.001 (`low_var_kl`), rollout n=16.122 123**RL leg 2 — tool calling (GRPO via verl)** — a second leg seeded from leg 1. The reward is the **official**124BFCL `ast_checker` (`bfcl_eval`). Each prompt renders its tool schemas through the model's **own** chat template125(byte-exact to the official `tools=` rendering), and the prompt set is filtered to only rows the official checker126can grade. 8-GPU wall, KL 0.001, rollout n=16; this repository ships the **step-40** checkpoint, selected for the127best BFCL / abstention balance and least policy drift. BFCL rose **76.8/71.0/69.5 → 85.0/84.0/80.5** with128instruction following, grounded abstention and knowledge all held.129 130**Forgetting control** — out-of-distribution broad-holdout perplexity at **0.99× the pre-RL base** (8.88 vs 8.97),131measured on a web crawl postdating the training corpus. No measurable forgetting.132 133## The medical foundation134 135**This is a medical model.** KOS-V5-Instruct inherits a base trained on a **54-source medical/biomedical corpus**136— not a general-purpose model with medical fine-tuning bolted on.137 138The strongest evidence is **bits-per-byte on held-out medical text**, which is tokenizer-agnostic and therefore139the only strictly fair cross-model comparison. In a **17-model pool** — including dedicated biomedical140specialists BioMedLM (300B PubMed tokens), Meditron-7B, PMC-LLaMA-7B and MedGemma-4B — the KOS-V5 base ranks141**first**:142 143| medical text (BPB, lower is better) | KOS-V5-Base | rank |144| :-- | --: | --: |145| **5-corpus mean, held-out medical text** | **0.4635** | **1 / 17** |146| clinical narratives | **0.4179** | **1 / 17** |147| radiology | **0.5132** | **1 / 17** |148| chest X-ray reports | **0.6688** | **1 / 17** |149| BIOSSES biomedical sentence similarity (Pearson / Spearman) | **0.7097 / 0.7014** | **1 / 17** |150| BLURB biomedical probe mean | 0.7268 | 2 / 17 |151 152Every comparator in that pool was trained on **1.3–153× more data** (0.3–36T tokens vs our 0.235T). See153[KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) for the full 96-metric evaluation.154 155### Medical MMLU (from the 57-subject run above)156 157The 9 medical subjects of MMLU, extracted from the same official 5-shot run:158 159| medical subject | KOS-V5-Instruct | KOS-V4-Instruct |160| :-- | --: | --: |161| high-school biology | 0.5774 | 0.2387 |162| clinical knowledge | 0.5623 | 0.3170 |163| nutrition | 0.5359 | 0.2843 |164| college biology | 0.5347 | 0.2917 |165| medical genetics | 0.5100 | 0.2700 |166| anatomy | 0.4815 | 0.3185 |167| professional medicine | 0.4375 | 0.2132 |168| college medicine | 0.4046 | 0.2486 |169| virology | 0.3795 | 0.2952 |170| **medical-9 mean** | **0.4915** | **0.2752** |171 172**+21.6 points over KOS-V4-Instruct**, and above the model's own full-MMLU average (0.4512) — the medical173subjects are where it is strongest.174 175### Medical QA benchmarks (official suites)176 177Official `medqa_4options`, `medmcqa` and `pubmedqa` from the same pristine lm-evaluation-harness,1785-shot, loglikelihood, metric `acc`.179 180| medical benchmark | **KOS-V5-Instruct** | KOS-V4-Instruct | YuLan-Mini-Instruct | marin-8b-instruct | Qwen3-4B-Instruct-2507 |181| :-- | --: | --: | --: | --: | --: |182| params | **3.7B** | 3.0B | 2.4B | 8.0B | 4.0B |183| **PubMedQA** | **0.7060** | 0.6860 | 0.6960 | 0.7500 | 0.7720 |184| MedQA (USMLE, 4-option) | **0.3802** | 0.2820 | 0.3511 | 0.4878 | 0.6159 |185| MedMCQA | **0.3648** | 0.2778 | 0.3856 | 0.4961 | 0.5804 |186 187**KOS-V5-Instruct improves on KOS-V4-Instruct on all three** (+9.8 MedQA, +8.7 MedMCQA, +2.0 PubMedQA).188 189**PubMedQA is the standout: 0.7060**, ahead of YuLan-Mini and within reach of Stanford's Marin-8B at **less than190half the parameters**. PubMedQA tests comprehension of **biomedical literature** — the closest of these three to191what the base was actually trained on. The USMLE-style exam MCQs (MedQA, MedMCQA) are where the answer-letter192bottleneck below bites hardest.193 194> ⚠️ **Why the MCQ numbers understate this model.** Our own measurements show KOS models place very little195> probability mass on MCQ answer *letters*: the format, not the knowledge, is the bottleneck. A model that ranks196> **1 of 17** at modelling clinical text while scoring modestly on multiple-choice is exhibiting exactly that gap.197> **Read the BPB results as the medical signal and the MCQ results as a floor, not a ceiling.**198 199## Evaluation & benchmarks200 201**Official suites only**, EleutherAI lm-evaluation-harness `0.4.12.dev0` at upstream commit `c1c4bea`, run from a202**pristine clone** with stock, unmodified task definitions.203 204- **IFEval** — stock `ifeval` task, 0-shot, greedy (`do_sample=false`, `temperature=0.0`),205  task-default `max_gen_toks=1280`, `--apply_chat_template`, seed 0. Constraint checking by the harness's vendored206  **Google** verifier (`instructions_registry`, 25 instruction types).207- **MMLU** — stock `mmlu` group, official 57 subjects / 14,042 test items, 5-shot from `dev` (`first_n`),208  loglikelihood over A–D, metric `acc` (not `acc_norm`), **no chat template**.209- **Medical QA** — stock `medqa_4options`, `medmcqa`, `pubmedqa` tasks, 5-shot, loglikelihood, metric `acc`,210  no chat template.211 212| benchmark | KOS-V5-Instruct | KOS-V4-Instruct | Δ |213| :-- | --: | --: | --: |214| **IFEval strict-avg** | **72.19** | 61.6 | **+10.6** |215| IFEval prompt-strict | 0.6728 | 0.5471 | +0.126 |216| IFEval inst-strict | 0.7710 | 0.6655 | +0.106 |217| IFEval prompt-loose | 0.6932 | 0.5693 | +0.124 |218| IFEval inst-loose | 0.7878 | 0.6882 | +0.100 |219| **MMLU (57-subj, 5-shot, `acc`)** | **0.4512** | 0.2782 | **+0.173** |220 221### Tool / function calling — official BFCL222 223Measured with the **official `bfcl_eval`** suite in **FC (function-calling) mode**, non-live categories, the model224prompted in its own native tool format and served via vLLM. **Tool calling is a trained objective of this model**225— the second GRPO leg optimised the official BFCL AST checker directly.226 227| BFCL (official, FC mode, **non-live AST**) | **KOS-V5-Instruct** | Qwen3-4B-Instruct-2507 (peer) | KOS-V4-Instruct |228| :-- | --: | --: | --: |229| simple *(334/400)* | **85.00** | 83.20 | 72.75 |230| multiple *(157/200)* | **84.00** | 79.00 | 73.00 |231| parallel *(147/200)* | **80.50** | 73.50 | 60.50 |232 233**KOS-V5-Instruct is above the Qwen3-4B-Instruct-2507 peer on all three BFCL categories**, and far above the234previous KOS-V4-Instruct. This is the axis the tool-calling GRPO leg was built to move, and it moved.235 236> **Scope.** These are the **non-live AST** categories only (`simple_python`, `multiple`, `parallel`). The237> live, multi-turn, web-search and memory categories were **not run**, so no BFCL *overall* score is reported238> here — the suite's aggregate column is not meaningful when most categories are unrun.239 240> **Engine note.** These BFCL numbers come from the official `bfcl_eval` harness on a **vLLM** backend, whereas241> the IFEval and MMLU figures on this card come from the HuggingFace backend of a pristine lm-evaluation-harness.242> Both are official suites; they are not the same inference stack, and that is stated rather than blurred.243 244**Cross-harness reproduction.** IFEval strict-avg measured **72.19** (pristine HF harness) and **72.0** (our245RL-evaluation harness) in two independent runs — a **0.19-point** agreement across two harness builds, far below246the benchmark's own ±2.14-point standard error on 541 prompts, so they are the same measurement.247 248**Harness validation.** The identical pipeline scored the peer mark **Qwen3-4B-Instruct-2507 at 84.71** IFEval249strict-avg on the same pristine harness, and independently reproduced KOS-V4-Instruct's MMLU to four decimal250places (0.2782). A score of 0.0 on this pipeline would therefore be a model property, not a harness failure.251 252### Grounded abstention & robustness — official RGB253 254Measured on the **official RGB harness** (retrieval-augmented generation benchmark).255 256| RGB (official) | KOS-V5-Instruct | Qwen3-4B-Instruct-2507 (peer) |257| :-- | --: | --: |258| **negative rejection** (declines the unanswerable) | **57.33** | 39.0 |259| noise robustness | 64.0 | 93.67 |260 261**Grounded abstention is a genuine strength: neg-reject 57.33 vs the peer's 39.0** — this model declines to answer262unanswerable questions far more often than it invents an answer. Noise-robustness (64.0) improved over an earlier263revision (58.67) but remains below the 70 threshold we treat as a pass.264 265### IFEval in context (strict-avg)266 267| model | weights | company | params | IFEval strict-avg |268| :-- | :-- | :-- | :-- | --: |269| GPT-4o-mini | Proprietary | OpenAI | 8B + | 79 \* |270| Llama-3.2-3B-Instruct | Open | Meta | 3.2B | 73.9 |271| **KOS-V5-Instruct (ours)** | **Open** | **Univ. of Kentucky / Louisville** | **3.7B** | **72.19** |272| Qwen2.5-3B-Instruct | Open | Alibaba | 3.0B | 64.7 |273| Phi-3-medium-4k-instruct | Open | Microsoft | 14.0B | 64.2 |274| Mistral-Large | Proprietary | Mistral AI | 46.7B + | 63 \* |275| **KOS-V4-Instruct (previous gen)** | **Open** | **Univ. of Kentucky** | **3.0B** | **61.6** |276| Yi-1.5-9B-Chat | Open | 01.AI | 8.8B | 60.5 |277| Phi-3.5-mini-instruct | Open | Microsoft | 3.8B | 57.7 |278| GPT-3.5-turbo-0613 | Proprietary | OpenAI | 20B + | 57 \* |279| Phi-3-mini-4k-instruct | Open | Microsoft | 3.8B | 56.1 |280| GPT-3.5-turbo-1106 | Proprietary | OpenAI | 20B + | 55.9 |281| Mistral-7B-Instruct-v0.2 | Open | Mistral AI | 7.2B | 55.0 |282| Llama-3.1-8B-Instruct | Open | Meta | 8.0B | 44.3 |283| Llama-2-13b-chat | Open | Meta | 13.0B | 39.8 |284 285**\* strict estimate** — no official IFEval strict sub-metrics published; estimated from published AVG4 or286prompt-strict (loose metrics run ~2–4 pts above strict). **\+ unofficial params.**287 288### Against university-built instruction models289 290Measured by us on the identical pristine harness, same protocol:291 292| model | institution | params | IFEval strict-avg | MMLU |293| :-- | :-- | --: | --: | --: |294| **KOS-V5-Instruct (ours)** | **UK / UofL** | **3.7B** | **72.19** | **0.4512** |295| marin-8b-instruct | Stanford | 8.0B | 70.83 | 0.6112 |296| YuLan-Mini-Instruct | Renmin | 2.4B | 61.51 | 0.5278 |297| **KOS-V4-Instruct (ours)** | **UK** | **3.0B** | **60.63** | 0.2782 |298| LLäMmlein-7B-chat | Würzburg | 7.0B | 54.07 | 0.5252 |299| Poro-34B-chat | U Turku | 34.2B | 34.63 | — |300| Minerva-7B-instruct | Sapienza | 7.4B | 21.51 | 0.4071 |301| CroissantLLMChat | CentraleSupélec | 1.3B | 19.94 | 0.2401 |302| Tucano-2b4-Instruct | U Bonn | 2.4B | 14.95 | 0.2589 |303 304On instruction following KOS-V5-Instruct now places **first among nine university-built instruct models**, ahead of305Stanford's Marin-8B (70.83) at less than half its parameters, and of Poro-34B at **9× its parameter count**. Note306that several of these models are non-English-first (Finnish, Italian, French, Portuguese, German) and are being307measured on English benchmarks, which understates their designed capability. On parametric knowledge (MMLU) the308larger, more heavily trained models still lead.309 310## Retrieval & embeddings311 312Beyond generation, KOS-V5-Instruct **also serves as a dense text retriever.** A companion **LoRA adapter** —313[**KOS-V5-Retriever**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever) — converts314this model into an embedding model (llm2vec-style: bidirectional attention + mean-pooling + a contrastively-trained315rank-32 LoRA), with these base weights **frozen and unchanged**.316 317On the **official BEIR SciFact** benchmark (the `beir` library + pytrec_eval — the public-leaderboard scorer),318**zero-shot** (training excluded SciFact, verified clean), it scores **NDCG@10 = 0.7007** (Recall@10 0.864) — a319**strong dense retriever**, above BM25 (0.665) and within the GTR/E5/BGE band (0.70–0.76). The adapter is320hot-swappable: attach it for retrieval, detach it for generation. See the321[adapter card](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever) for the encode recipe and322the full retrieval details.323 324## Known regressions and limitations325 326> These are disclosed deliberately. A high benchmark score does not make this checkpoint production-ready.327 328- **Tool / function calling is a trained strength — and now measured against the peer.** The second GRPO leg329  optimised the *official* BFCL AST checker directly; BFCL is 85.0/84.0/80.5, above the peer and far above the330  previous KOS generation. Earlier internal KOS tool-calling figures are deliberately omitted: several were331  measured against benchmark data the model had been trained on and are recorded in our own audit as **invalid**.332- **Grounded abstention / fabrication IS measured — and strong.** On the official RGB harness, negative-rejection333  is 57.33 (peer 39.0). *Earlier revisions of this card stated fabrication was NOT independently measured; it now334  is.* Noise-robustness (64.0) is still below our 70 pass bar, so retrieval-noise handling remains a known gap.335- **The peer leads on knowledge and raw instruction following.** IFEval 72.19 vs the peer's 84.71; MMLU 0.4512 vs336  0.7266; medical QA below the peer. This is a from-scratch 3.7B model on 235B tokens against one trained on orders337  of magnitude more data — strong for its scale, not state-of-the-art in absolute terms.338- **MMLU 0.4512 is above chance (0.25) but modest.** This is not a knowledge model; it should not be used as a339  medical question-answering authority.340- **Not a medical-MCQ model.** As with KOS-V4, do not benchmark or deploy it as one; read the BPB results (base341  card) as the medical signal and the MCQ results as a floor.342 343## Data contamination344 345- **IFEval: CLEAN (verbatim).** The SFT mix and the IFEval RL prompt set were exact-containment scanned against346  IFEval's official 541 test prompts — **0 exact containments**. Exact matching cannot detect paraphrase or347  reformatting.348- **Tool calling (BFCL): CLEAN (verbatim).** The tool-calling RL prompt pool was exact-containment scanned against349  **5,437 full-length official BFCL prompts — 0 exact containments**.350- **MMLU / PubMedQA / MedQA / MedMCQA: UNCHECKED.** Contamination against these four has **not** been scanned for351  this checkpoint. Those numbers should be read with that caveat.352 353## Prompt / chat format (ChatML)354 355```356<|im_start|>system357You are a helpful assistant.<|im_end|>358<|im_start|>user359Write a haiku about Kentucky. Do not use any commas.<|im_end|>360<|im_start|>assistant361```362 363Tool / function calling uses the model's native `<tools>` … `</tools>` schema block and `<tool_call>` … `</tool_call>`364response format; pass your function schemas via the tokenizer's `apply_chat_template(..., tools=[...])`.365 366## Quickstart367 368```python369from transformers import AutoModelForCausalLM, AutoTokenizer370 371model_id = "Kentucky-Open-Science/KOS-V5-Instruct"372tok = AutoTokenizer.from_pretrained(model_id)373model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")374 375messages = [{"role": "user", "content": "Write a haiku about Kentucky. Do not use any commas."}]376inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)377out = model.generate(inputs, max_new_tokens=256, do_sample=False)378print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))379```380 381## Deployment382 383| Precision | Approx. VRAM | Notes |384| :--- | :--- | :--- |385| **bfloat16** | ~9 GB | native weights (7.43 GB) + activations; a single 16 GB GPU is comfortable |386 387## Related models388 389- [**KOS-V5-Retriever**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever) — a LoRA390  **retrieval/embedding adapter** for this model (llm2vec; official BEIR SciFact NDCG@10 **0.70**, zero-shot).391- [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) — the from-scratch pretrained392  foundation this model is tuned from (3.72B, 235.2B tokens).393- [**KOS-V4-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct) — previous generation394  (3.0B, IFEval 61.6), the public release.395- [**KOS-V4-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Base) — previous-generation foundation396  (3.015B, 180.3B tokens).397 398## Intended use & limitations399 400Research use only. English only. Not for clinical, commercial, legal, or production-grade use. Outputs may be401factually wrong or fabricated. This model must not be used to make or inform medical decisions.402 403## Naming404 405The program is **KOS** (KOS-V1..V6). Earlier internal names are not used.406