WindyITS/business-model-kg-query-planner-data

Business Model KG Query Planner Dataset

This repository contains the curated training and evaluation data used for the Business Model KG local query stack.

The query stack has two jobs:

• —a router decides whether a user question can be answered locally, should fall back to a stronger hosted model, or should be refused
• —a planner converts supported local questions into compact query plans that the project runtime can compile into read-only Cypher

This dataset is the supervision source for that workflow.

Layout

train.jsonl
validation.jsonl
release_eval.jsonl
planner_only_open_literal_copying_augmentation.jsonl
train_synthetic_graphs.json
validation_synthetic_graphs.json
release_eval_synthetic_graphs.json
manifest.json
checksums.txt
curation_log.jsonl

Splits

• —train.jsonl: 8,000 examples
• —validation.jsonl: 1,200 examples
• —release_eval.jsonl: 1,800 examples
• —planner_only_open_literal_copying_augmentation.jsonl: 90 targeted planner augmentation examples

The router and planner did not use every file in the same way. The DeBERTa router was fine-tuned on the 8,000-example training split. The 1,200-example validation split was kept separate for model selection and threshold calibration: the runtime does not always take the highest-scoring router class, but applies calibrated local and refuse thresholds so uncertain questions fall back to the hosted path. release_eval.jsonl was kept separate as a held-out release evaluation split.

The 90-row planner-only augmentation was added after the DeBERTa router was already fine-tuned. It was not used to retrain the router. It exists because the Qwen planner needed a small amount of targeted extra supervision for open literal-copying behavior, and rerunning the router fine-tune was unnecessary.

The synthetic graph files provide companion graph contexts for the corresponding splits. checksums.txt, manifest.json, and curation_log.jsonl are included so the release can be audited.

Router Threshold Calibration

The DeBERTa router outputs scores for three labels: local, api_fallback, and refuse. The runtime does not simply choose the highest-scoring label. Instead, it applies a calibrated policy:

if local score >= local threshold:
  route to the local Qwen planner
elif refuse score >= refuse threshold:
  refuse the question
else:
  fall back to the hosted path

The thresholds were computed on the 1,200-example validation split after router fine-tuning. First, the trained router scored the validation questions. Then temperature scaling was fitted on the validation logits to calibrate the output probabilities. The local threshold was chosen first by searching validation scores for the cutoff that reached the target local precision while preserving as much recall as possible. After applying that local gate, the refuse threshold was chosen on the remaining validation examples with the same precision-first logic.

In the final bundle, the target precision constraints were 97% for local and 95% for refuse. Questions that do not pass either threshold are routed to api_fallback.

Format

Rows are JSONL objects. Important fields include:

• —question: the user-facing natural-language question
• —supervision_target: the main training target used by the export scripts
• —route_label: the router label
• —gold_cypher, gold_params, gold_rows: reference artifacts for evaluation and inspection
• —metadata: provenance and curation metadata

Training code should consume supervision_target, because it makes local_safe, strong_model_candidate, and refuse explicit in the serialized target.

Intended Use

Use this dataset to reproduce or extend the Business Model KG query router and planner training workflow. It is also useful for inspecting what kinds of questions the local runtime is expected to answer.

Limitations

The dataset is specialized for the ontology, Cypher contract, and runtime constraints of Business Model KG. It should not be treated as a general-purpose question-answering dataset. Some fields are designed for internal validation and may not transfer cleanly to other graph schemas.