davidafrica/gloss

gloss: derived data from five protocols on natural-language descriptions of Qwen3-8B

Version 1.0.0. Public derived-data release of five completed protocols that asked one question: can a short natural-language description S of Qwen3-8B, learned by a proposer/search loop, help a frozen predictor forecast what the model does next (its next reasoning step, or its next response-mode change) better than the prompt and the model's own partial output alone, and better than data-free control texts of the same length?

This is not a raw archive and not a standalone benchmark. It contains labels, label counts, predictor probabilities, per-item losses, split/pool membership, lineage identifiers, the reviewed description texts and run provenance. It contains no prompts, chains of thought, continuations, trajectories, question wording or options, token ids, embeddings, API transcripts or judge rationales. Multiple-choice letters keep their meaning only relative to the withheld option order of each question.

Loading

from datasets import load_dataset
ds = load_dataset('davidafrica/gloss', 'mode_time_predictions', split='test', revision='v1.0.0', token=False)
print(ds[0]['condition'], ds[0]['predictor'], ds[0]['logloss'])
runs = load_dataset('davidafrica/gloss', 'runs', split='records', revision='v1.0.0', token=False)

Every configuration is a plain Parquet table (no loading script, no trust_remote_code). Configuration names are <protocol>_<table kind> plus the two shared tables runs and descriptions.

Configurations

Split semantics

• —pilot_mcq_examples: train/test are the pilot's own prompt-level split (400/98 chat, 400/100 math prompts); neither is a training set for anything released here
• —pilot_mcq_predictions: same prompt-level split as the examples
• —contrastive_mcq_examples: train (1,590 kept development questions, later reused by corrected_search) / test (374 held-out questions)
• —contrastive_mcq_predictions: same prompt-level split; the development split holds headroom scores only
• —corrected_search_membership: dev = the contrastive_mcq train questions partitioned into feedback / acceptance / selection groups
• —corrected_search_examples: fresh_test only
• —corrected_search_predictions: selection (development selection pool) / fresh_test
• —blind_methods_examples: frozen pools: feedback, acceptance, gepaval, finalsel, test
• —blind_methods_predictions: frozen pools; only test carries the six evaluated conditions
• —mode_time_prefixes: frozen pools: feedback, acceptance, gepaval, finalsel, test, xstest (benign reserve)
• —mode_time_futures: same pools as the prefixes
• —mode_time_predictions: same pools; external and base-rate forecasts exist for test and xstest only
• —descriptions: records
• —runs: records

The five protocols are not independent test sets. corrected_search reuses the contrastive_mcq development questions (same question_id, cut_id, prompt_id values) for its feedback / acceptance / selection pools; contrastive_mcq cuts the same 998 chains as pilot_mcq (shared prompt_id values). contrastive_mcq_predictions rows with stage = diagnostic_retrospective are a researcher-approved retrospective re-scoring of an already examined test set, not a fresh confirmation. corrected_search rows with condition = S_8b_prev score the previous run's final description as a diagnostic comparator. Nothing was re-clustered or re-split for this release.

Identifiers and lineage

All identifiers are keyed pseudonyms (p_ prompt, c_ cut, q_ question, g_ group, f_ family, i_ forecast item). The same underlying object receives the same pseudonym everywhere it appears, so joins between examples, membership, predictions, futures and prefixes work within and across protocols where the source records establish identity. The original-identifier crosswalk is retained privately by the authors and is not part of this release. Content fingerprints (item_hash) are withheld.

Probability and label semantics

• —probs for predictor = qwen3-8b are raw first-token probability masses over the option letters (A-D) or the sixteen outcome codes (A-P); letter_mass / code_mass is their sum before any normalisation. Values are preserved at source precision (float64) and were not renormalised.
• —anthropic/claude-sonnet-5 rows carry a parsed letter (pred, pred_common) in the MCQ protocols and elicited probabilities in mode_time; pred = null means the answer did not parse and was scored wrong.
• —acc is 1 when pred equals correct. In contrastive_mcq, acc_8b / acc_14b score the prediction against each model's own modal label.
• —mode_time losses use the pre-registered smoothing p' = (p + 1e-6) / (1 + 16e-6), mean of -log p'(outcome) over the labelled futures (logloss); event_logloss and timing_logloss decompose it. outcome_codes are the ten realised codes of the item.
• —A completion event means the response ended without a further mode change; it is not a refusal. The next event is not the final compliance status of the response, and P (survival) means no new event within the 2,048-token window, not that the model never refuses.
• —Outcome labels in runs are the scientist label (outcome_label_scientist, the curated verdict) and the worker label recorded at the time (outcome_label_worker); both are provenance, not new verdicts.

See CODEBOOK.md for every column and enumeration and LICENSES.md for rights.

Coverage (original vs released)

Descriptions: every text that entered a scored condition or the retrospective diagnostic is released after review (descriptions.text_released); unselected search candidates, rejected edits and the per-round search histories are withheld because they are unreviewed free text tied to no reported score. The truncated run-1 mode_time text is included as superseded_run1 and was never scored on the test pool.

Limitations

• —pilot_mcq stopped at its headroom check; its predictions test the instrument, not any description.
• —contrastive_mcq used one development pool for feedback, acceptance and selection; its held-out contrasts did not separate from controls.
• —corrected_search detected a below-target gain (+2.71 points) without model specificity; blind_methods found no learned-vs-nothing gain surviving Holm.
• —mode_time passed its registered joint log-loss test, but all Qwen conditions are worse than uniform and a base-rate table beats every model predictor; treat the gain as calibration and timing repair.
• —Labels were produced by LLM annotators (openai/gpt-5.6-terra, anthropic/claude-sonnet-5); agreement diagnostics are reported in the source analyses, not re-derived here.
• —mode_time request families come from WildJailbreak (gated, ODC-BY) and XSTest; only numerical/label derivatives and opaque identifiers are released, so the rows cannot be mapped back to particular requests from this dataset alone.

Provenance

Source experiment identifiers and final commits are in the runs configuration and in release_manifest.json (sha256 of every released file). Numeric fields were copied from the source score and label files without recomputation and verified value-for-value after writing.

Citation

@dataset{africa2026gloss,
  author = {Africa, David Demitri},
  title = {gloss: derived data from five protocols on natural-language descriptions of Qwen3-8B},
  year = {2026}, version = {1.0.0}, publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/davidafrica/gloss}
}