ajaxdavis/donto-qwen3.8-27b-predicate-extractor

Donto-Qwen3.8-27B Predicate Extractor

An experimental, anchor-free, abundance-oriented predicate-extraction LoRA for short English document chunks.

This repository contains the selected V15 adapter for `Qwen/Qwen3.8-27B`, pinned to base revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. It turns a general Qwen3.8 language backbone into a specialist that emits validated Donto fact objects instead of conversational prose.

The headline result is 89.21% exact recall and 89.44% exact precision over a frozen, source- and template-disjoint, 100-document graph-first synthetic gold suite containing 5,338 target facts. All 2,718 forced structured calls succeeded.

This is a research release, not a claim that open-world extraction is solved. Read What the score does and does not mean and Limitations before deploying it.

The ambition

Knowledge systems were designed for a world in which producing structured claims was scarce. Generative models invert that constraint: candidate properties and relations can be produced in abundance. The new problem is not merely generating one summary or one canonical graph. It is holding a large, multi-directional, possibly contradictory field of claims without erasing disagreement.

Donto is built for that second problem. It is a contradiction-preserving claim substrate: emit free predicates now; preserve separate claims; defer ontology alignment, entity resolution, joining, standing, and reconciliation to later query-time processes. Donto-Qwen is the extraction front door. Its job is to read a bounded source unit carefully and produce many compact candidate claims that the larger Donto system can hold, compare, cite, align, and re-rank.

The purpose of this adapter is deliberately narrower than general chat:

1. 1.recover atomic subject–predicate–object relations;
2. 2.preserve attribution, negation, modality, qualification, and disagreement;
3. 3.represent direct claims separately from derived or hypothetical views;
4. 4.use free, concise predicates instead of forcing every source into a fixed ontology;
5. 5.stop cleanly when a bounded extraction task is exhausted;
6. 6.return only a validated tool call—never prose around hand-parsed JSON.

What this model is

This adapter changes the base model's behaviour, not its underlying factual knowledge. It does not intrinsically make Qwen3.8 decode faster. Its efficiency advantages come from short inputs, compact structured outputs, shared prompt prefixes, fewer retries, and greater document concurrency.

Output contract

Every call must produce the submit_facts tool with this shape:

{
  "facts": [
    {
      "s": "ex:review-panel",
      "p": "postponed",
      "o": "ex:deployment",
      "c": 0.99,
      "h": false
    },
    {
      "s": "ex:postponement-wasted-money",
      "p": "contrastsWith",
      "o": "ex:postponement-protected-staff",
      "c": 0.92,
      "h": true
    }
  ],
  "more_supported_facts": false
}

• —s: subject IRI, normally a stable ex:kebab-case identifier.
• —p: concise camelCase free predicate, with standard RDF predicates allowed.
• —o: entity IRI or string, numeric, or Boolean literal.
• —c: model confidence in [0, 1]; this is not calibrated probability.
• —h: false for direct/source-licensed claims; true for a derived, interpretive, hypothetical, or otherwise non-direct view.
• —more_supported_facts: continuation signal for the current bounded task.

The exact OpenAI-compatible tool declaration is in `schemas/tools.json`.

Extraction architecture used for the reported scores

The benchmark did not ask one vague prompt to "extract everything." Each document was traversed through explicit bounded tasks:

This decomposition is important. A single generic extraction prompt is not expected to reproduce the published multi-stage score.

Results

Headline evaluations

Gold 100 also produced:

• —4,762 exact matches from 5,338 targets;
• —5,324 valid predicted facts;
• —zero failed calls;
• —zero malformed fact objects;
• —zero duplicate exact items;
• —two reflexive entity errors.

Gold 100 by task

The overall release clears the project's predeclared 80% aggregate-gold target, but the direct D0 slice does not. That distinction is intentional and public. The weakest blind families were long wetlands argument/procurement documents; the strongest were bounded incidents, metadata, residual stopping, and structural relations.

Full row-level outputs—not summaries alone—are published under `evaluation/`.

What the score does and does not mean

The metric uses one-to-one exact keys over (s, p, o, h). A supported free predicate synonym or stable entity alias receives no credit, which can make the score conservative. Conversely, the gold documents are programmatically generated from explicit graphs and templates. They measure controlled compositional transfer, not the full ambiguity, OCR damage, genre diversity, or factual uncertainty of unseen real-world documents.

The 100-document suite was generated after V15 training, then frozen and hashed before the first V15 inference. It has:

• —100 unique document hashes;
• —73 parent templates;
• —zero source-hash overlap with the 240 V15 training/validation documents;
• —zero parent-template overlap with the 102 training/validation templates;
• —45–59 unique gold facts per document (mean 52.88).

The run used one seed. It is not a multi-seed reliability estimate and is not a human-blind real-document benchmark. The prior real llm_nerd fixture was heavily used for development diagnosis and is therefore not presented as blind evidence.

Training data

All distributable training, validation, validator, gold, manifest, and audit files are published at `ajaxdavis/donto-qwen3.8-27b-predicate-extraction-data`.

Canonical V15 training mixture:

• —4,255 training rows;
• —9,013 target facts;
• —200 source families;
• —4,123 synthetic task rows from 152 generated documents;
• —132 reviewed-real rows from 48 source families;
• —1,085 validation rows with 2,140 facts from 40 source- and template-disjoint synthetic documents;
• —no training/validation source-family overlap;
• —no training/validation parent-template overlap;
• —every rendered row below the 4,096-token contract.

The dataset card separates synthetic and reviewed-real provenance and explains its mixed licensing status.

Training recipe

Hybrid-layer LoRA coverage

Qwen3.8-27B contains 48 Gated DeltaNet layers and 16 full-attention layers. A generic q_proj/v_proj LoRA recipe would miss most recurrent projections. This adapter explicitly targeted 496 language modules / 992 LoRA tensors and no vision module:

• —48 each: linear_attn.in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj;
• —16 each: self_attn.q_proj, k_proj, v_proj, o_proj;
• —64 each: mlp.gate_proj, up_proj, down_proj.

The complete name-level audit is `artifacts/lora-target-audit.json`.

Serving with vLLM on one RTX 3090

The physically tested path used vLLM 0.27.1, Transformers 5.15.0, bitsandbytes 0.50.1, a 4,096-token limit, four concurrent sequences, and prefix caching. vLLM required the tensor-preserving conditional-key remap under `runtime/vllm-conditional/`; the root adapter remains the canonical native PEFT artifact.

hf download ajaxdavis/donto-qwen3.8-27b-predicate-extractor \
  --local-dir ./donto-qwen

vllm serve Qwen/Qwen3.8-27B \
  --served-model-name qwen3.8-27b \
  --language-model-only \
  --quantization bitsandbytes \
  --load-format bitsandbytes \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --generation-config vllm \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-lora \
  --lora-modules donto-qwen=./donto-qwen/runtime/vllm-conditional \
  --max-loras 1 \
  --max-lora-rank 16

The exact tested launcher includes additional GPU-specific cache and attention flags and is preserved at `training/source/start_runpod_quality_canary.sh`.

Then call the forced tool and validate its arguments with a real schema. A complete Pydantic/OpenAI-compatible example is in `examples/structured_client.py`.

Native PEFT use

The root adapter_model.safetensors is the canonical PEFT adapter. Load it over the exact pinned base revision with current Transformers and PEFT. The training script in `training/source/train_qwen38_donto_3090.py` contains the memory-safe text-only 3090 loader and exact QLoRA configuration.

Do not attach the root native adapter directly to vLLM if its Qwen3.8 key resolver expects the conditional language-model root; use the verified remap supplied in runtime/vllm-conditional.

Efficiency profile

• —Adapter weights are about 467 MB; the base model remains the dominant memory and compute cost.
• —The adapter does not improve raw decode tokens/second by itself.
• —Short chunks reduce prefill and KV-cache costs.
• —Prefix caching reuses the large common tool/prompt prefix across documents.
• —The anchor-free five-field format reduces output tokens per useful fact.
• —Four concurrent sequences fit in the physically tested 24 GB 3090 serving path.
• —Gold 100 completed 2,718 structured calls in 8,511 seconds at concurrency four, about 19.2 bounded decisions per minute. This is not a generic tokens/second benchmark.

Limitations

1. 1.Direct extraction is the weakest blind slice. Gold-100 D0 exact recall and precision are both about 75%.
2. 2.Synthetic-domain bias. The main quantitative evidence comes from generated graph/template documents, not a large human-authored blind corpus.
3. 3.One training seed. The recipe has not been repeated across multiple seeds.
4. 4.Two reflexive errors occurred in Gold 100. Role direction and reflexivity still require downstream validation.
5. 5.No evidence anchors. This release deliberately omits source spans. A separate citer or human review is required when provenance is important.
6. 6.Free predicates are not ontology alignment. Predicate and entity aliases may vary; alignment belongs downstream.
7. 7.Confidence is uncalibrated. Do not interpret c as a probability of truth.
8. 8.`h:true` is not established fact. It marks derived or interpretive output.
9. 9.Not a contradiction resolver. It may emit incompatible claims by design; Donto preserves and later ranks them.
10. 10.Not for unattended high-stakes use. Legal, medical, financial, historical, or personal claims require source review.
11. 11.Not a general assistant. Fine-tuning trades conversational versatility for structured extraction behaviour.
12. 12.Multimodal behaviour is unevaluated. No vision module was adapted.

Development history and negative results

This release is the fifteenth curriculum iteration, trained fresh from the pinned stock base. Rejected adapters were never reused as weight bases. The repository preserves rejection reports for V11–V14 because the failures are informative:

• —V11 learned structured calls but missed its precision gate and had semantic contamination in one generator family.
• —V12 completed finitely but was too terse and failed real-document coverage.
• —V13 reached strong source-disjoint exact metrics but retained role/scope errors.
• —V14 nearly memorized its supervised contract (about 99% exact) yet transferred poorly to the real development document because its curriculum rewarded short, clean pages.
• —V15 started fresh and taught dense D0 pages, nonempty recovery passes, explicit exhaustion, larger local D1 views, and broader graph families.

See `docs/` for the plan, audits, launch receipt, runbook, and complete negative-result reports.

Reproducibility and integrity

• —Native final adapter SHA-256: 40d11bcad20061f8291741c4dd8a3701405902f336c27b60592d62d6b2f7a974
• —vLLM remapped adapter SHA-256: 28df663ffa70bf541cafff2c9bb60df11e4fe000a2ea9124b346a9cc3eb20d8b
• —Training corpus SHA-256: 1b3630d00d11476c7f1e941a60a442a5eb5eb57dd882b7d188ef9b85d461e738
• —V15 validation SHA-256: c360e8296e5ec7973f2fab8416d0b30e2aa850fe6b2cc618bb3fa52894d818b2
• —Gold-100 dataset SHA-256: 6f16c87d06c2a0b850321b8750e54a63364937cfa2823f34f698f6c00f160f1c
• —Gold-100 evaluation SHA-256: a21e32e6db21f217af4ad7fc66cbcb3836f0607567bdaab9fcc5c11859e2fd55

The repository includes intermediate adapter checkpoints 100–500, trainer states, the exact trainer/evaluator/generator sources, training and serving logs, run contracts, target-module audit, and full row-level evaluation outputs. Unsafe and unnecessary optimizer/RNG pickle states are not part of the public release.

The Hub upload, clean-download hash checks, and unauthenticated browser proof are recorded in `PUBLICATION-RECEIPT-20260825.md`.

Licence and acknowledgements

The adapter and code are released under Apache-2.0, consistent with the pinned Qwen3.8-27B base. Dataset licensing is documented separately because the corpus contains both generated material and a small reviewed-real research tier.

Built on Qwen3.8-27B, PEFT, Transformers, bitsandbytes, Liger Kernel, and vLLM. The practical RTX 3090 serving work was informed by `syv-ai/qwen38-27b-rtx3090`.

Citation

@misc{donto_qwen38_v15_2026,
  author       = {Davis, Thomas},
  title        = {Donto-Qwen3.8-27B Predicate Extractor V15},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/ajaxdavis/donto-qwen3.8-27b-predicate-extractor}},
  note         = {Experimental LoRA adapter for structured Donto predicate extraction}
}