project-telos/doorkey-semantic-reasoning-labels

GPT-OSS-20B DoorKey semantic reasoning labels

This dataset contains automatic sentence-level semantic-function annotations for 7,038 reasoning sentences produced by openai/gpt-oss-20b on 46 fixed DoorKey environment states. Each target sentence is paired with its preceding reasoning context and assigned one or more human-readable discourse labels.

The annotation run produced 7,036 valid rows and two schema failures. These are model-generated exploratory annotations, not human gold labels.

An independent full-corpus replicate produced 7,037 valid rows. Among the 7,036 sentences valid in both runs, exact multi-label agreement is 74.9% and derived primary-label agreement is 85.4% (Cohen's kappa 0.820). These measure repeatability, not accuracy against human labels.

Labels

• —state_readout
• —route_planning
• —verification
• —correction
• —new_inference
• —consolidation
• —restatement
• —procedural_continuation
• —action_commitment

Definitions, boundaries, and legitimate overlaps are provided in taxonomy/MULTILABEL_TAXONOMY.md.

Configurations

annotations

The complete judge output. Important fields include:

• —annotation_id, sentence_id: stable sentence identifiers;
• —context_before, target_sentence: input shown to the judge;
• —semantic_labels: JSON-encoded multi-label set;
• —primary_label: deterministic precedence-derived grouping label;
• —four observable cue fields;
• —annotation_status, confidence, rationale;
• —judge, prompt, token-usage, retry, raw-response, and error provenance.

primary_label is derived from semantic_labels; it is not an independent human or second-judge annotation.

sentence_inventory

The source inventory before annotation, including stable IDs, environment and sentence positions, preceding context, and target sentences.

annotations_replicate

An independent second annotation of the same inventory with the same model, prompt, taxonomy, seed, temperature, and reasoning effort. Pair it with annotations on annotation_id to reproduce the agreement diagnostics.

Loading

from datasets import load_dataset

annotations = load_dataset(
    "project-telos/doorkey-semantic-reasoning-labels",
    "annotations",
)
replicate = load_dataset(
    "project-telos/doorkey-semantic-reasoning-labels",
    "annotations_replicate",
)
inventory = load_dataset(
    "project-telos/doorkey-semantic-reasoning-labels",
    "sentence_inventory",
)

The semantic_labels CSV field can be decoded with json.loads.

Provenance

• —Judge: Together AI openai/gpt-oss-20b
• —Reasoning effort: low
• —Temperature: 0
• —Prompt version: general_multilabel_v3_human_calibrated
• —Prompt, schema, input, and few-shot SHA-256 hashes are preserved under metadata/.
• —The exact few-shot demonstrations are included under metadata/.
• —Original and replicate manifests and status files are included separately under metadata/.

Limitations

• —Labels are automatic and imperfect. They should support exploratory stratification and example retrieval, not be treated as settled ground truth.
• —The independent full-corpus replicate measures production repeatability but not correctness or human agreement.
• —new_inference remains broad and often overlaps verification or route planning.
• —Confidence is highly concentrated at high and is not empirically calibrated.
• —Fine boundaries such as route planning versus action commitment remain ambiguous.
• —Two rows contain errors instead of valid semantic labels.
• —The reasoning traces come from a single model family and DoorKey task.

See reports/V3_FULL_AUDIT_REPORT.md for descriptive corpus statistics and reports/V3_FULL_REPLICATE_DIAGNOSTICS.md for agreement definitions, trajectory-bootstrap intervals, per-label stability, and disagreement examples.

License

Apache-2.0. Users should also comply with the terms applicable to the source trajectory dataset and GPT-OSS model.