Berk/assay-synthetic
assay-synthetic Models trained on this: the Assay collection -- six decision models from 149M to 27B. Berk/assay-4b is the usual choice; Berk/assay-compiled-base runs on a CPU. Synthetic decision tasks generated by the Assay project. Each item is a state (the facts), a typed question (bool, choice or score with described options) and a label computed in code, so the labels are exact rather than annotated. They were written to fix two measured weaknesses of decision models:… See the full description on the dataset page: https://huggingface.co/datasets/Berk/assay-synthetic.
assay-synthetic
Models trained on this: the Assay collection -- six decision models from 149M to 27B. Berk/assay-4b is the usual choice; Berk/assay-compiled-base runs on a CPU.
Synthetic decision tasks generated by the Assay project. Each item is a state (the facts), a typed question (bool, choice or score with described options) and a label computed in code, so the labels are exact rather than annotated. They were written to fix two measured weaknesses of decision models: composing rules and reasoning about dates. On our public transfer suite, adding them to training moved deadline accuracy from 0.60 to 0.975 and rule-composition families by 3 to 12 points.
Format
One JSON object per line:
{"state": {"policy": "...", "case": "..."},
"questions": {"verdict": {"type": "score", "instructions": "How late is the report?",
"levels": ["On time", "Late but accepted", "Refused"], "label": 1}},
"meta": {"source": "dates", "id": "dates/0"}}state is a string or an object of named fields. Each question carries its type and its options: options (a map from option key to description) for choice, levels (ordered, low to high) for score, optional yes/no descriptions for bool. label is the option key, a boolean or a level index. answerable is false when the state is missing a fact the rule needs, which is the case in part of policy_hard: a model should say so rather than guess. Some items carry target, a probability distribution over the options, when the label is soft.
The rubrics.json file holds the 90 generic questions (50 bool, 20 choice, 20 score) that Assay uses to label unlabelled text with a teacher model; they are reusable as a distillation or evaluation rubric bank.
Uses
Training or evaluating models that must apply rules exactly, follow nested conditions, refuse when information is missing, or compare dates written in different formats. The verdicts are computed, so the test splits are reliable as an evaluation of that capability, and a model that has not seen the generators cannot have memorised them.
Generation
Everything here is produced by code in the Assay repository (assay/data/policy.py, assay/data/dates.py, assay/data/rubrics.py); no text is taken from any other dataset. Regenerate at any size with a different seed:
from assay.data.policy import generate, generate_hard
from assay.data.dates import generate as generate_dates
examples = generate(10_000, seed=0) + generate_hard(10_000, seed=1) + generate_dates(10_000, seed=2)The models trained on it
The Assay collection holds the models this data was generated for: four decoder sizes from 0.6B to 27B and an encoder tier, each answering typed questions with calibrated probabilities, an evidence score and conformal abstention. python -m assay.pipeline trains one on data in this format.
Licence and scope
Apache-2.0. This repository contains only generated material. Assay also trains on public datasets which are not redistributed here; they keep their own licences, several of them non-commercial, and are listed with their sources in docs/datasets.md.