CoolFace
Datasetpublic

ClarusC64/reasoning-constraint-loss-attribution-v0.1

Reasoning Constraint Loss Attribution v0.1 A SIOS research dataset for identifying when a governing constraint ceases to regulate a reasoning trajectory, locating the first point of loss, attributing the lost constraint, and identifying the structural mechanism that produced the loss. Repository: ClarusC64/reasoning-constraint-loss-attribution-v0.1 Version: 0.1.0 Publisher: Clarus Invariant Framework: SIOS Dataset identity Reasoning Constraint Loss Attribution… See the full description on the dataset page: https://huggingface.co/datasets/ClarusC64/reasoning-constraint-loss-attribution-v0.1.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes46downloads
Dataset Card

Reasoning Constraint Loss Attribution v0.1

A SIOS research dataset for identifying when a governing constraint ceases to regulate a reasoning trajectory, locating the first point of loss, attributing the lost constraint, and identifying the structural mechanism that produced the loss.

Repository:

text
ClarusC64/reasoning-constraint-loss-attribution-v0.1

Version:

text
0.1.0

Publisher:

text
Clarus Invariant

Framework:

text
SIOS

Dataset identity

Reasoning Constraint Loss Attribution v0.1 is a structured reasoning dataset.

It is not a single-label classification task.

The model must determine:

  1. 1.whether a governing constraint was lost;
  2. 2.where the loss first occurred;
  3. 3.which supplied constraint was lost;
  4. 4.which structural mechanism family produced the loss.

The central question is:

Which governing constraint ceased to regulate the reasoning trajectory, where did that transition begin, and what structural mechanism caused it?

The dataset includes a reproducible scorer, but it is not presented as a statistically mature benchmark or formal leaderboard.

Its current purpose is to establish a clear task definition, annotation structure, prediction contract, and evaluation method for constraint-loss attribution.


Relationship to Reasoning Drift Onset Detection v0.2

Reasoning Drift Onset Detection v0.2 evaluates a broader reasoning state:

text
outcome mode
drift onset
mechanism family
repair outcome
residual constraint

Reasoning Constraint Loss Attribution v0.1 isolates a narrower problem:

text
constraint-loss detection
first loss-step localisation
lost-constraint attribution
mechanism-family attribution

It does not evaluate:

text
outcome_mode
repair_outcome
correction_source
residual_constraint_failure
recommended_repair
repair_instruction

This narrower structure makes the dataset suitable for focused training and evaluation of constraint attribution without requiring the full repair and outcome ontology.


Repository structure

text
reasoning-constraint-loss-attribution-v0.1/
├── data/
│   ├── train.csv
│   └── test.csv
├── scorer/
│   └── score.py
├── README.md
├── CITATION.cff
└── LICENSE

Split sizes

SplitRecordsLabels included
train.csv24Yes
test.csv12Yes

Total labelled records:

text
36

The public test split is intended for reproducible evaluation and pipeline testing.

Because the gold test labels are public, this release cannot prevent test-set adaptation.


Data structure

The labelled CSV files contain the following fields:

text
scenario_id
domain
task
constraints_json
trajectory_json
constraint_loss
first_loss_step
lost_constraint
mechanism_family
loss_mechanism
visible_consequence_step
severity
trajectory_length
relative_loss_position
trajectory_structure
annotation_confidence

These fields are divided into four groups:

text
Model inputs
Primary targets
Secondary annotations
Audit metadata

Model inputs

The model receives:

text
scenario_id
domain
task
constraints_json
trajectory_json

scenario_id

A unique identifier for each reasoning scenario.

Example:

text
RCLA-025

domain

The reasoning domain represented by the scenario.

The current release includes examples from:

text
clinical reasoning
scientific reasoning
legal reasoning
engineering
machine learning
policy analysis
ecology
operations
cybersecurity
financial analysis
education
historical analysis
software debugging

The domain field supports analysis.

The current dataset is not large enough to support reliable domain-specific competence estimates.

task

A natural-language description of the reasoning problem.

Example:

text
Assess whether recurrent headaches are explained by one reported trigger.

constraints_json

A JSON array containing the governing constraints the reasoning trajectory should preserve.

Example:

json
[
  "Preserve multiple plausible causes until discriminating evidence appears.",
  "Separate temporal association from established causation.",
  "Do not treat one recurring factor as sufficient explanation."
]

Each constraint expresses a condition that must continue to govern the reasoning process.

Constraint loss occurs when one of these conditions ceases to regulate the trajectory operationally, even if the constraint remains mentioned descriptively.

trajectory_json

A JSON array containing ordered reasoning steps.

Example:

json
[
  {
    "step": 1,
    "text": "Headaches often occur after poor sleep."
  },
  {
    "step": 2,
    "text": "They also occur after missed meals and during periods of high stress."
  },
  {
    "step": 3,
    "text": "Poor sleep is selected as the primary cause because it appears most consistently."
  }
]

Trajectory steps must:

  • begin at 1;
  • use contiguous numbering;
  • contain non-empty text;
  • preserve their original order;
  • contain exactly the fields step and text.

Primary targets

The dataset defines four primary targets.

1. constraint_loss

A Boolean label indicating whether a supplied governing constraint was lost.

Valid values:

text
true
false

true

At least one supplied constraint ceases to regulate the reasoning trajectory.

The lost constraint may still be mentioned after the loss point, but it no longer materially constrains the conclusion, decision, or path of analysis.

false

All supplied constraints remain operationally active throughout the trajectory.

A trajectory may:

  • rank one hypothesis above another;
  • reduce uncertainty;
  • prioritise one test;
  • apply proportionate caution;
  • reach a qualified conclusion;

without losing a governing constraint.

2. first_loss_step

The earliest step at which the selected governing constraint is no longer operationally preserved.

For no-loss cases, the value is blank.

The first loss step should identify the structural transition, not merely the later point where an explicit unsupported conclusion or action becomes visible.

Example:

text
first_loss_step = 3
visible_consequence_step = 5

This means the constraint stopped governing the reasoning at Step 3, while the visible consequence did not appear until Step 5.

3. lost_constraint

The exact text of the first governing constraint lost.

For loss cases, the value must exactly match one item in constraints_json.

Example:

text
Preserve multiple plausible causes until discriminating evidence appears.

For no-loss cases:

text
none

The labelled dataset stores the full constraint text for interpretability.

Prediction files use a 1-based constraint identifier instead.

4. mechanism_family

The coarse structural mechanism that produced the constraint loss.

Valid values:

text
none
premature_constraint_closure
scope_collapse
causal_scope_collapse
multi_causal_collapse
multi_objective_collapse
proxy_substitution
evidence_suppression
evaluation_scope_collapse

none is valid only for no-loss cases.


Mechanism-family definitions

premature_constraint_closure

The reasoning closes uncertainty, a differential, a hypothesis set, or a decision branch before sufficient discriminating evidence is available.

Typical forms include:

text
anomaly to confirmed state
compatible evidence to diagnosis
single observation to forecast
association to primary cause

scope_collapse

A conclusion expands beyond the scope supported by the available evidence, procedure, authority, or observation.

Typical forms include:

text
notice to liability
behaviour to character trait
necessary condition to sufficient condition
procedural compliance to complete legitimacy

causal_scope_collapse

An association, temporal sequence, comparison, adjusted result, or visible intervention is treated as supporting more causal certainty than the evidence permits.

Typical forms include:

text
correlation to causation
adjustment to causal certainty
temporal proximity to causal priority
single event to system-wide effect

multi_causal_collapse

An interacting causal structure is reduced to one dominant cause, event, catalyst, or pathway.

Other causes may remain mentioned, but they lose operative explanatory force.

Typical forms include:

text
catalyst to primary cause
narratively vivid event to sole explanation
visible intervention to dominant cause

multi_objective_collapse

Several decision objectives remain visible descriptively, but one objective captures the actual decision process.

Typical competing objectives include:

text
average benefit
distributional fairness
risk
cost
implementation uncertainty
resilience
long-term adaptation

proxy_substitution

A proxy, surface measure, procedural artifact, or partial indicator replaces the broader state it was intended to represent.

Typical forms include:

text
signature to informed consent
profit to financial resilience
vegetation cover to ecosystem recovery
single species response to ecosystem recovery

evidence_suppression

Missing, weak, conflicting, or unresolved evidence is converted into support for closure or removed from active reasoning.

Typical forms include:

text
missing logs to successful operation
missing telemetry to normal operation
absence of recorded failure to absence of failure

evaluation_scope_collapse

Performance under a limited evaluation setting is treated as evidence of broader generalisation, robustness, safety, or deployment readiness.

Typical forms include:

text
validation gain to generalisation
benchmark success to safety
aggregate accuracy to deployment readiness

Secondary annotations

The labelled dataset includes:

text
loss_mechanism
visible_consequence_step
severity

These fields support analysis but are not official prediction targets in v0.1.

loss_mechanism

A fine-grained diagnostic subtype within the broader mechanism family.

Example:

json
{
  "mechanism_family": "causal_scope_collapse",
  "loss_mechanism": "adjustment_to_causation"
}

The coarse mechanism family is the official prediction target.

Fine-grained mechanisms remain exploratory because many currently have only one example.

visible_consequence_step

The first step where constraint loss produces an explicit unsupported:

text
conclusion
classification
recommendation
decision
action
attribution

This may occur after first_loss_step.

The field may be blank where constraint loss remains latent and does not yet produce a visible consequence.

For no-loss cases, the field is blank.

severity

A coarse estimate of the consequence of the loss within the designed scenario.

Valid values:

text
none
low
medium
high

Severity is scenario-relative.

It should not be interpreted as a universal risk scale or as a domain-calibrated safety measure.


Audit metadata

The labelled files include:

text
trajectory_length
relative_loss_position
trajectory_structure
annotation_confidence

These fields support dataset auditing, shortcut analysis, and stratified evaluation.

They should not be treated as model inputs.

trajectory_length

The number of objects in trajectory_json.

relative_loss_position

Calculated as:

text
first_loss_step / trajectory_length

Example:

text
first_loss_step = 3
trajectory_length = 5
relative_loss_position = 0.600

The field is blank for no-loss cases.

trajectory_structure

A descriptive label for the designed trajectory geometry.

Examples include:

text
stable_hypothesis_ranking
linear_closure_to_action
branch_suppression_without_explicit_falsehood
aggregate_capture_of_distribution
evaluation_scope_expansion
narrative_capture_of_multicausality

This is audit metadata rather than a prediction target.

annotation_confidence

The dataset author’s confidence in the annotation.

This field does not replace:

text
independent annotation
expert adjudication
inter-annotator agreement

State consistency rules

A valid no-loss case requires:

text
constraint_loss = false
first_loss_step = blank
lost_constraint = none
mechanism_family = none

A valid loss case requires:

text
constraint_loss = true
first_loss_step = positive integer
lost_constraint = exact supplied constraint
mechanism_family = non-none family

The loss step must not exceed the trajectory length.

The lost constraint must resolve to exactly one item in constraints_json.


Prediction format

Prediction files must contain:

csv
scenario_id,predicted_constraint_loss,predicted_first_loss_step,predicted_lost_constraint_id,predicted_mechanism_family

Example:

csv
scenario_id,predicted_constraint_loss,predicted_first_loss_step,predicted_lost_constraint_id,predicted_mechanism_family
RCLA-025,true,3,1,premature_constraint_closure
RCLA-026,false,,none,none

predicted_constraint_loss

Valid values:

text
true
false

predicted_first_loss_step

A positive integer indicating the predicted first loss step.

For predicted no-loss cases, leave the field blank.

predicted_lost_constraint_id

A 1-based index into constraints_json.

Example:

json
[
  "Preserve multiple plausible causes until discriminating evidence appears.",
  "Separate temporal association from established causation.",
  "Do not treat one recurring factor as sufficient explanation."
]

A prediction of:

text
1

selects:

text
Preserve multiple plausible causes until discriminating evidence appears.

For predicted no-loss cases, use:

text
none

or leave the field blank.

predicted_mechanism_family

The predicted coarse mechanism family.

For predicted no-loss cases:

text
none

Prediction files must contain exactly one row for every gold scenario.

The scorer rejects:

  • missing predictions;
  • extra predictions;
  • duplicate scenario identifiers;
  • unknown scenario identifiers;
  • invalid Boolean values;
  • invalid mechanism-family labels;
  • invalid constraint identifiers;
  • loss steps outside the trajectory;
  • no-loss predictions containing attribution values;
  • loss predictions missing attribution values.

Evaluation design

The scorer separates two different evaluation questions:

text
End-to-end performance
Conditional attribution performance

This distinction is necessary because no-loss examples do not contain a real loss step or lost constraint to localise.

A correct no-loss prediction should receive credit for detecting that no loss occurred.

It should not receive additional attribution credit for matching:

text
None == None

on the loss step or lost constraint.


End-to-end metrics

End-to-end metrics are calculated across every scenario.

They evaluate whether the model handles both loss and no-loss trajectories correctly.

Constraint-loss macro F1

Macro F1 across:

text
false
true

This is the primary detection metric.

It gives equal weight to the loss and no-loss classes regardless of class frequency.

Complete Attribution Accuracy

A case receives credit only when all four primary targets are exactly correct:

text
constraint_loss
first_loss_step
lost_constraint_id
mechanism_family

Formula:

text
Complete Attribution Accuracy =
Number of cases with all four targets correct
/
Total number of evaluated cases

For a no-loss case, a complete correct prediction requires:

text
constraint_loss = false
first_loss_step = blank
lost_constraint_id = none
mechanism_family = none

For a loss case, all four loss-attribution fields must match exactly.

This is a strict end-to-end reconstruction metric.


Conditional attribution metrics

Conditional attribution metrics are calculated only on scenarios where:

text
gold constraint_loss = true

No-loss cases receive no credit and do not enter the denominator.

These are the primary metrics for localisation and attribution quality.

Exact first-loss-step accuracy

Measures exact agreement on first_loss_step among gold loss cases.

Formula:

text
Correct exact loss-step predictions
/
Number of gold loss cases

A blank onset prediction receives no credit on a gold loss case.

Within-one first-loss-step accuracy

Measures whether the predicted loss step is within one step of the gold onset.

For numerical predictions:

text
absolute error <= 1

A blank onset prediction receives no credit.

Lost-constraint exact accuracy

Measures whether the model selects the correct 1-based lost-constraint identifier among gold loss cases.

No-loss examples are excluded.

Conditional mechanism-family macro F1

Evaluates mechanism-family attribution only on gold loss cases.

The none class is excluded.

The fixed loss-family label space is:

text
premature_constraint_closure
scope_collapse
causal_scope_collapse
multi_causal_collapse
multi_objective_collapse
proxy_substitution
evidence_suppression
evaluation_scope_collapse

This metric isolates mechanism attribution from loss detection.

A model does not receive mechanism-attribution credit merely for predicting none correctly on no-loss examples.

Mean absolute first-loss-step error

Calculated only on gold loss cases where the model supplies a numerical first-loss-step prediction.

Formula:

text
sum of absolute onset errors
/
number of gold loss cases with numerical onset predictions

Because blank onset predictions are excluded from the MAE calculation, the scorer also reports:

text
numeric_first_loss_step_prediction_coverage

This prevents a model from appearing to have low onset error by omitting difficult predictions.


Secondary end-to-end diagnostics

The scorer retains several all-case diagnostics for transparency:

text
first_loss_step_exact_accuracy_all_cases
lost_constraint_exact_accuracy_all_cases
mechanism_family_macro_f1_including_none

These are not the primary attribution metrics.

The first two include correct None/None matches on no-loss cases.

They may therefore be inflated by strong no-loss detection.

The scorer output includes an explicit warning:

All-case localisation and constraint metrics include correct None/None matches on no-loss cases and must not be used as the primary attribution measures.

End-to-end mechanism-family macro F1

The scorer reports an all-case mechanism-family macro F1 using:

text
none
plus all eight loss families

This metric combines:

text
loss detection
mechanism attribution

It is useful as an end-to-end diagnostic but should not replace conditional mechanism-family macro F1.


Macro F1 policy

All macro F1 metrics use fixed predefined label spaces.

This means that a class with zero gold support still appears in the macro calculation.

In the current scorer:

text
zero-support classes receive an F1 value of 0

and therefore contribute zero to the macro average.

This policy has two benefits:

  • results remain comparable across runs using the same fixed taxonomy;
  • missing class coverage is not hidden.

It also means that macro F1 can be low on a small test set even when predictions are correct for all represented classes.

For that reason, every macro F1 result must be interpreted alongside class-support counts.


Support counts

The scorer reports:

text
loss_case_support
no_loss_case_support

These identify the denominators of the two evaluation layers.

It also reports class support for:

text
constraint_loss
mechanism_family_all_cases
mechanism_family_loss_cases

Example:

json
{
  "loss_case_support": 8,
  "no_loss_case_support": 4
}

Conditional localisation and attribution metrics use only loss_case_support.


Scorer output

The scorer returns a JSON report with the following structure:

json
{
  "dataset": {
    "name": "Reasoning Constraint Loss Attribution v0.1",
    "evaluation_type": "structured_constraint_attribution",
    "n": 12,
    "loss_case_support": 8,
    "no_loss_case_support": 4
  },
  "end_to_end_metrics": {
    "constraint_loss_macro_f1": 0.8333,
    "complete_attribution_accuracy": 0.5,
    "complete_attribution_required_targets": [
      "constraint_loss",
      "first_loss_step",
      "lost_constraint_id",
      "mechanism_family"
    ]
  },
  "conditional_attribution_metrics": {
    "evaluation_subset": "gold constraint_loss=true cases only",
    "support": 8,
    "exact_first_loss_step_accuracy": 0.625,
    "within_one_first_loss_step_accuracy": 0.875,
    "lost_constraint_exact_accuracy": 0.75,
    "mechanism_family_macro_f1": 0.5417,
    "mechanism_labels": [
      "premature_constraint_closure",
      "scope_collapse",
      "causal_scope_collapse",
      "multi_causal_collapse",
      "multi_objective_collapse",
      "proxy_substitution",
      "evidence_suppression",
      "evaluation_scope_collapse"
    ],
    "mean_absolute_first_loss_step_error": 0.7143,
    "numeric_first_loss_step_prediction_coverage": 0.875
  },
  "secondary_end_to_end_diagnostics": {
    "first_loss_step_exact_accuracy_all_cases": 0.75,
    "lost_constraint_exact_accuracy_all_cases": 0.8333,
    "mechanism_family_macro_f1_including_none": 0.5926
  },
  "class_support": {},
  "per_class_metrics": {},
  "metric_policy": {
    "fixed_label_macro_f1": true,
    "zero_support_classes_contribute_zero": true,
    "conditional_attribution_subset": "gold constraint_loss=true",
    "none_excluded_from_conditional_mechanism_f1": true,
    "negative_cases_receive_no_conditional_attribution_credit": true
  },
  "prediction_table": []
}

Per-scenario prediction table

The scorer includes a per-scenario audit table.

Each row contains:

text
gold targets
predicted targets
loss-detection correctness
end-to-end field correctness
conditional attribution inclusion
conditional step correctness
conditional lost-constraint correctness
absolute onset error
complete attribution correctness

For no-loss cases:

text
conditional_attribution.included = false

The conditional attribution fields are returned as null.

This makes the distinction between detection and attribution explicit at the scenario level.


Running the scorer

Run the scorer with:

bash
python scorer/score.py \
  data/test.csv \
  predictions.csv

Save the JSON report:

bash
python scorer/score.py \
  data/test.csv \
  predictions.csv \
  --output results/score_report.json

Interpretation guidance

The evaluation should be read in layers.

Layer 1 — Loss detection

Use:

text
constraint_loss_macro_f1

This asks:

Can the model distinguish trajectories that preserve their constraints from trajectories that lose one?

Layer 2 — Conditional attribution

Use:

text
exact_first_loss_step_accuracy
within_one_first_loss_step_accuracy
lost_constraint_exact_accuracy
conditional mechanism-family macro F1

These ask:

Given that constraint loss genuinely occurred, can the model locate and explain it?

Layer 3 — Complete end-to-end reconstruction

Use:

text
complete_attribution_accuracy

This asks:

Can the model produce the complete four-field attribution state correctly across both loss and no-loss cases?

A model may perform well on detection while performing poorly on attribution.

A model may also show reasonable component performance while rarely reconstructing the complete attribution state.

These differences are analytically meaningful and should remain visible.


Why no-loss cases are excluded from conditional attribution

A no-loss example has no real:

text
first loss step
lost constraint
loss mechanism

When both gold and prediction contain blank values, the match is valid as part of the complete end-to-end state.

It is not evidence that the model can localise or attribute an actual loss.

For this reason:

text
None == None

is counted in Complete Attribution Accuracy but excluded from the primary conditional localisation and attribution metrics.

This prevents strong no-loss prediction from artificially improving apparent attribution performance.


Recommended scorer tests

The scorer should be tested against at least the following cases before release:

Perfect predictions

Expected result:

text
all applicable metrics = 1.0

Zero-support mechanism families may still reduce fixed-label macro F1 unless every fixed family is represented in the test fixture.

All-no-loss predictions

Expected result:

  • no-loss examples may be correct;
  • gold loss examples must fail detection;
  • conditional localisation and attribution metrics must receive no credit from no-loss examples;
  • blank predicted onsets on gold loss cases must count as incorrect;
  • numerical onset coverage should fall.

Onset off by one

Expected result:

text
exact onset = incorrect
within-one onset = correct
MAE = 1

Correct detection, wrong mechanism

Expected result:

  • constraint-loss detection correct;
  • complete attribution incorrect;
  • conditional mechanism attribution incorrect.

Correct detection, wrong constraint

Expected result:

  • detection correct;
  • lost-constraint attribution incorrect;
  • complete attribution incorrect.

Missing scenario identifier

Expected result:

text
scorer exits with missing prediction error

Extra scenario identifier

Expected result:

text
scorer exits with extra prediction error

Duplicate prediction identifier

Expected result:

text
scorer exits with duplicate identifier error

Out-of-range loss step

Expected result:

text
scorer rejects a step greater than trajectory length

Invalid constraint identifier

Expected result:

text
scorer rejects an index outside constraints_json

Invalid mechanism family

Expected result:

text
scorer rejects an unknown mechanism-family label

Zero-support mechanism families

Expected result:

  • unsupported fixed classes remain present;
  • their support is 0;
  • their precision, recall, and F1 are 0;
  • they contribute zero to fixed-label macro F1.

Current dataset composition

The current release contains:

text
24 training scenarios
12 test scenarios
36 total scenarios

The examples include:

text
constraint-preserving trajectories
early loss
middle loss
late loss
latent structural narrowing
visible unsupported consequences
single-cause collapse
objective capture
proxy substitution
evidence suppression
evaluation overreach

The dataset deliberately includes stable controls that are close to failure cases.

These controls are intended to distinguish legitimate hypothesis ranking or qualified reasoning from actual constraint loss.


Current limitations

Small dataset

The dataset contains only 36 labelled trajectories.

This is sufficient for task definition, scorer development, and initial model probing.

It is not sufficient for stable model ranking.

Small test split

The public test split contains 12 scenarios.

Conditional attribution metrics may therefore have small denominators.

Sparse mechanism support

Several mechanism families have limited support.

Some fixed mechanism families may have zero examples in a given evaluation split.

Because fixed-label macro F1 assigns zero to unsupported classes, the metric must always be read alongside class-support counts.

Synthetic trajectories

The scenarios are designed examples rather than naturally generated model traces.

Synthetic design permits precise control of constraints and loss points, but it may also introduce regularities not present in real reasoning.

Explicit constraints

The governing constraints are supplied directly to the model.

The dataset does not currently evaluate whether a model can infer hidden or implicit constraints.

Single primary loss assumption

Each scenario identifies one primary first-lost constraint.

The dataset does not yet evaluate:

text
simultaneous constraint losses
competing loss attributions
loss cascades
constraint restoration
repeated loss

Domain breadth without domain depth

The dataset spans multiple reasoning domains.

Most domains contain too few scenarios for reliable domain-specific estimates.

Annotation subjectivity

The release does not include formal inter-annotator agreement.

The exact first-loss step and mechanism family may require adjudication in some cases.

Public test labels

The test labels are public.

The release supports reproducible evaluation but not protected leaderboard comparison.


Annotation roadmap

A future annotation study should use multiple independent annotators.

Recommended agreement measures:

FieldSuggested measure
Constraint-loss presenceFleiss’ kappa
First loss stepExact and within-one agreement
Lost constraintExact constraint-selection agreement
Mechanism familyFleiss’ kappa
Fine loss mechanismExploratory agreement
Visible consequence stepExact and within-one agreement
SeverityWeighted agreement

Expected agreement is likely to be highest for:

text
constraint-loss presence
lost-constraint selection

Moderate for:

text
first loss step
mechanism family
visible consequence step

Lowest for:

text
fine loss mechanism
severity
trajectory structure

Roadmap

A stronger future release should include:

  • more examples per mechanism family;
  • balanced loss and no-loss support;
  • more stable near-neighbour controls;
  • wider onset-position coverage;
  • three-step trajectories;
  • eight-to-twelve-step trajectories;
  • latent loss without visible consequence;
  • multiple simultaneous constraint losses;
  • repeated loss and restoration;
  • competing lost constraints;
  • cascading constraint failure;
  • natural model-generated traces;
  • adversarial lexical controls;
  • hidden-constraint attribution;
  • domain-held-out evaluation;
  • independent annotation;
  • expert adjudication;
  • inter-annotator agreement;
  • human baselines;
  • public model baselines;
  • protected test labels.

Recommended status

Version 0.1 should be treated as:

A structured research dataset and evaluation seed for constraint-loss detection, localisation, and attribution.

It should not yet be treated as:

  • a statistically mature benchmark;
  • a definitive model-ranking instrument;
  • a production safety evaluation;
  • evidence of domain-specific professional competence;
  • a protected public leaderboard.

Status

text
status: research_dataset
version: 0.1.0
task_definition_stable: true
evaluation_pipeline_included: true
benchmark_status: false
leaderboard_ready: false
conditional_attribution_metrics: true
negative_attribution_inflation_corrected: true
fixed_label_macro_f1: true
zero_support_classes_contribute_zero: true

Licence

MIT.

The dataset is synthetic and contains no third-party source data.


Citation

text
Caplan, M. (2026).
Reasoning Constraint Loss Attribution v0.1.
Clarus Invariant / SIOS.
Dataset.

See:

text
CITATION.cff

for machine-readable citation metadata.