barryzhang/ornith-harness-4b-v2

Ornith Harness 4B v2

Ornith Harness 4B v2 is a 4.2-billion-parameter, Qwen3.5-derived text model distributed as a Q5KM GGUF artifact for local, machine-verifiable data-processing and coding workflows. The release is designed for systems with approximately 16 GB of available accelerator or unified memory and includes the thin tool-use harness used for final evaluation. The training version had three fewer audit-only fields and produced the same model-visible messages.

This repository contains the frozen v2 research release. Its primary uses are local agent research, verifier-based coding workflows, and reproducible benchmark experiments. General chat evaluation, ranking against frontier models, and autonomous production deployment are outside the evaluated scope. A registered, single-use 48-task final evaluation was completed on 30 July 2026; optimization stopped after the results were read.

Highlights

• —Compact deployment artifact: 3.07 GB GGUF, evaluated with an 8,192-token context.
• —Independent final estimate: v2 passed 32/48 tasks, for pass@1 0.667 (Wilson 95% CI 0.525–0.783).
• —Within-lineage post-training gain: compared with C28, the pre-self-distillation initialization checkpoint, under the same Q5KM evaluation protocol, v2 improved from 8/48 to 32/48 tasks passed, a descriptive gain of 50.0 percentage points on this holdout.
• —Higher completion rate under the registered deployment protocol: under this specific fixed 8K harness, runtime, prompt, and token budget—but with unmatched quantization—v2 Q5KM passed 32/48 tasks and Ornith-1.0-9B Q4KM passed 10/48. This is a comparison of the two packaged deployment routes under the registered protocol.
• —Lower operational overhead: v2 had the lowest median primary tokens and wall time among the three final-evaluation arms.
• —Matched model and harness: the release includes the two-tool agent loop and the evaluation entry points used to produce the reported development results.
• —Reproducible identity: model, harness, Modelfile, and evaluation artifacts are bound to SHA256 identifiers.

Model overview

Model lineage

The public lineage is:

1. 1.Qwen3.5 4B architecture;
2. 2.supervised fine-tuning on successful Ornith-1.0-9B tool-use trajectories;
3. 3.C28 same-policy preference optimization, producing the pre-self-distillation initialization checkpoint;
4. 4.first-round on-policy self-distillation (v1);
5. 5.second-round on-policy self-distillation, with checkpoint-51 selected and formally merged into the v2 release artifact.

The intermediate C28 and self-distillation weights are internal lineage artifacts. This repository distributes the final v2 GGUF and the runtime materials required to evaluate it.

Intended use

Primary intended uses are:

• —local agent research in an isolated tool-execution environment;
• —verifier-based data-processing and coding workflows;
• —reproducible benchmark experiments with the included harness;
• —workflows where the user provides concrete input data, an explicit output format, and at least one representative input-output example when the transformation is ambiguous.

User acceptance validation is performed after delivery and is not part of the default model prompt. The included development evaluator uses executable checks to reproduce the reported research metrics. It does not define the production input contract.

Representative tasks include deterministic data normalization, aggregation, interval processing, state transformation, and generation of small executable scripts that operate on user-provided files.

Out-of-scope uses include:

• —general chat evaluation and general conversational assistance;
• —ranking against frontier models or general-purpose coding leaderboards;
• —repository-scale refactoring and open-ended software architecture;
• —unsupervised acceptance-criteria design and security-critical code generation;
• —autonomous production deployment.

Training

Ornith Harness 4B was trained through two on-policy self-distillation rounds. In this release, self-distillation refers to verifier-guided self-improvement: the current policy generates trajectories, an objective verifier filters successful samples or constructs preference pairs, and the resulting data are used to update the same model lineage through SFT and DPO.

Assistant-token masking excludes tool responses from the SFT loss. The v2 checkpoint was merged and converted with --no-mtp; the merge changed 128 of 426 tensors, consistent with the trainable projection layout.

Evaluation

Final holdout protocol

• —Dataset: 48 tasks isolated from checkpoint selection.
• —Sampling: one rollout per task and model; 144 episodes across three arms.
• —Arms: C28 Q5KM, Ornith Harness 4B v2 Q5KM, and Ornith-1.0-9B Q4KM.
• —Seed: 20260724; concurrency: 1.
• —Hardware: one NVIDIA RTX PRO 6000 Blackwell.
• —Runtime: the same Ollama service, harness, prompt, examples, output format, and token budget for all three arms.
• —Generation: temperature 0.6, top-p 0.95, top-k 20, 2,048 completion tokens per turn, and 12,000 primary completion tokens per episode.
• —Context: 8,192 tokens.
• —Primary success event: PASS_VERIFIED_STOP.
• —Intervals: task-level Wilson 95% intervals; paired comparisons use exact two-sided McNemar tests.

Final holdout results

C28 and v2 passed the same eight tasks; v2 alone passed 24 additional tasks. The paired difference was +0.500, with exact two-sided McNemar p=1.19e-7.

v2 and 9B both passed eight tasks; v2 alone passed 24 and 9B alone passed two. The 9B-minus-v2 difference was -0.458, with exact two-sided McNemar p=1.05e-5.

Under the fixed 8K context, episodes with history compaction were 37/48 for C28, 32/48 for v2, and 35/48 for 9B. Total compactions were 195, 113, and 142. These are operational endpoints, not causal estimates of compaction-induced failure.

Development selection protocol

• —Dataset: eight held-out development tasks covering CSV grouping, interval merging, module splitting, state simulation, and text normalization.
• —Sampling: five independent rollouts per task, 40 episodes per model.
• —Hardware: one NVIDIA RTX PRO 6000 Blackwell.
• —Runtime: Ollama, c28_harness.py, sequential execution, no rescue turns.
• —Generation: temperature 0.6, top-p 0.95, top-k 20, 2,048 completion tokens per turn, 12,000 primary completion tokens per episode.
• —Context: 8,192 tokens.
• —Primary success event: objective verifier satisfied (PASS_VERIFIED_STOP).

Development selection results

The v2 formal rerun produced 27 PASS_VERIFIED_STOP, 12 FAIL_TASK, and one FAIL_GENERATION_CAP outcome across 40 episodes.

pass@1 is the mean single-rollout success rate. pass@5 records whether each task had at least one successful rollout. Realizable BoN@5 selects a candidate using the task's agent-visible executable checks and then scores the selected output with the offline objective verifier. The BoN result therefore requires a validation rule and is not available for tasks without one.

These development results selected checkpoint-51 from seven candidates. The v2 value 0.675 is therefore a selection statistic subject to winner's curse, not the independent headline performance estimate. This repository does not include a production N=5 execution and selection component.

Comparison boundary

The 4B-to-9B comparison represents two artifacts designed for an approximately 16 GB single-device deployment constraint. Quantization is not matched: v2 uses Q5KM, while Ornith-1.0-9B uses Q4KM. Evaluation ran on an RTX PRO 6000, not on 16 GB hardware. The table therefore compares deployment routes under a fixed harness and budget; it does not rank the underlying models at equal precision or claim measured 16 GB-device performance.

Published Ornith-1.0-9B benchmark results use different weights, context lengths, agent frameworks, and compute budgets. They are not comparable with the local development protocol reported here.

Usage

Place ornith-harness-4b-v2-Q5_K_M.gguf in this directory, then create the Ollama model:

ollama create ornith-harness-4b-v2 -f Modelfile

Run the included development evaluation:

python eval_selfdistill.py \
  --whitelist dev_whitelist.json \
  --pool pool/dev8.jsonl \
  --url http://127.0.0.1:11434/v1/chat/completions \
  --model ornith-harness-4b-v2 \
  --label eval \
  --n-reps 5 \
  --concurrency 1 \
  --rescue-turns 0 \
  --out /tmp/ornith-harness-4b-v2-eval \
  --distill-root . \
  --harness-profile v2 \
  --max-tokens 2048 \
  --primary-completion-budget 12000

python bon_report.py /tmp/ornith-harness-4b-v2-eval/rows/eval.jsonl

The working directory must be writable and traversable by the sandbox identity that executes model-generated commands.

Safety and operational requirements

The harness executes model-generated shell commands and writes files. Run it only in an isolated workspace with:

• —no production credentials or personal data;
• —restricted network access;
• —explicit CPU, memory, process, file-size, and time limits;
• —a disposable filesystem or recoverable snapshot;
• —independent validation of every delivered output.

Do not execute untrusted model output directly on a host system.

Limitations

1. 1.The final estimate comes from a fixed 48-task engineering benchmark, not a random sample of all real user tasks.
2. 2.Results are bound to one machine, one runtime, one harness, and one decode configuration.
3. 3.Realizable BoN@5 requires an executable user-visible validation rule.
4. 4.The current harness stops immediately when the objective verifier passes. Consequently, successful episodes have no later turn in which to declare DONE; DONE precision is NOT_EVALUATED_DESIGN and self-reported completion is not a supported selector.
5. 5.The 4B and 9B comparison uses different quantization levels.
6. 6.The final comparison ran on an RTX PRO 6000. Actual 16 GB hardware deployment remains a separate acceptance test.
7. 7.BoN@5 is an offline development analysis in this release; a production multi-sample runtime and sandbox have not been delivered.
8. 8.This research release supports artifact verification and development-set evaluation. Reproducing the full training, checkpoint-selection, and sealed-holdout workflow requires controlled project assets and collaboration with the maintainers.

Artifact integrity

The release manifest SHA256SUMS.txt records the package-level file hashes.

Upstream references

• —Qwen3.5-4B model card
• —Ornith-1.0-9B model card

The upstream Ornith attribution, copyright notice, and MIT License text are preserved in `NOTICE`.

When citing this release, include the repository revision and the v2 GGUF SHA256 listed above.