ClarusC64/clinical-intervention-sequencing-and-state-control-v0.1

# Clinical Multi-Evidence State Integration Benchmark

**Version:** 0.1.0  
**Short name:** CMESI  
**Release status:** Locked v0.1  
**Data type:** Synthetic  
**Language:** English  
**Licence:** MIT

## Overview

The Clinical Multi-Evidence State Integration Benchmark evaluates whether a language model can reconstruct how the state of a clinical system changes as evidence arrives over time.

Most clinical reasoning benchmarks ask a model to identify a diagnosis, select an answer, or predict a final state.

CMESI asks a more demanding question:

> Given a prior clinical state and an ordered sequence of evidence, can the model determine which tracked items changed, which remained stable, when each transition occurred, and which evidence was responsible?

Each scenario contains multiple tracked items such as:

- diagnoses;
- investigations;
- treatments;
- referrals;
- constraints.

Evidence arrives sequentially. The model must update each item selectively rather than treating the scenario as a single classification problem.

The benchmark therefore evaluates:

- final-state reconstruction;
- intermediate state tracking;
- evidence-order sensitivity;
- transition timing;
- selective preservation;
- competing diagnoses;
- evidence override;
- process resolution;
- reactivation;
- decisive-evidence identification;
- minimal decisive evidence;
- matched-pair consistency.

## Benchmark hypothesis

A clinically reliable reasoning system should:

1. update each tracked item only when justified;
2. preserve items unaffected by the evidence;
3. distinguish supportive from confirmatory evidence;
4. distinguish weakening from exclusion;
5. respect the order in which evidence arrives;
6. identify the event responsible for each decisive transition;
7. reconstruct the path leading to the final state;
8. remain internally consistent across all structured outputs.

A model that reaches the correct final state through an incorrect trajectory has not fully reconstructed the case.

## Important safety statement

CMESI is a synthetic research benchmark.

It does not contain real patient records, protected health information, or clinical cases copied from medical files.

It must not be used for:

- real-patient diagnosis;
- treatment selection;
- autonomous clinical decision-making;
- replacement of qualified medical judgement;
- direct patient care.

Strong benchmark performance does not establish clinical safety or deployment readiness.

## Repository structure

## Dataset composition

The complete release contains:

The split is:

Both members of every matched pair remain in the same split.

This is essential because pair members are deliberately contrastive and often differ by only one clinically meaningful feature.

## Domains

### Training split

The training data covers:

* respiratory medicine;
* infectious disease;
* emergency medicine;
* vascular medicine;
* gastroenterology;
* endocrinology;
* cardiology;
* dermatology;
* nephrology;
* neurology;
* psychiatry;
* geriatrics;
* pulmonology.

### Test split

The held-out test data covers:

* rheumatology;
* ophthalmology;
* hepatology;
* geriatric medicine;
* allergy and immunology;
* sleep medicine;
* orthopaedics;
* haematology.

The test split includes domains not represented directly in the training split, helping evaluate whether the model learns the benchmark’s state-integration structure rather than memorising domain-specific cases.

## Matched-pair design

CMESI uses contrastive matched pairs.

A pair typically preserves:

* the prior context;
* tracked items;
* initial states;
* much of the evidence structure.

It then changes one or more decisive evidence events.

This allows the benchmark to test whether the model responds to the evidence difference rather than reproducing a generic answer.

Matched pairs evaluate mechanisms such as:

or:

A model should solve both members of the pair to receive full pair-level credit.

## Core task

The model receives:

The model must return:

No natural-language explanation is required for scoring.

## Input structure

A simplified input record looks like this:

## Required prediction structure

A prediction record must contain:

## Tracked-item types

The v0.1 release includes:

Item identifiers use short prefixes:

The prefix is descriptive rather than logically determinative. Scoring is based on the item identifier and gold structure.

## State ontology

The v0.1 state ontology is:

### active_suspected

The item remains a live possibility but is not confirmed.

### active_confirmed

The item is established or operationally active.

### downgraded

The item remains possible but has been weakened.

### ruled_out

The item is no longer supported within the benchmark scenario.

### pending

An investigation or workflow remains incomplete.

### resolved

The investigation, referral, or active process has reached its annotated endpoint.

### inactive

A treatment, constraint, or process is not active.

### historical

The item is no longer active but remains relevant as a past event or exposure.

### superseded

A previous formulation has been displaced by a better-supported alternative.

## Transition ontology

The v0.1 transition ontology is:

### no_change

The state remains unchanged after the evidence event.

### confirm

A suspected item becomes confirmed.

### downgrade

The evidence weakens the item without excluding it.

### rule_out

The item moves to `ruled_out`.

### reactivate

A downgraded, ruled-out, or inactive item returns to an active confirmed state.

### resolve

A pending or active workflow reaches its endpoint.

### supersede

A formulation is displaced by another explanation.

### deactivate

An active treatment or constraint becomes inactive.

### mark_historical

An inactive or previously active item becomes explicitly historical.

## Evidence-role ontology

Evidence is annotated separately for every tracked item.

The same event may play different roles for different items.

The v0.1 evidence-role ontology is:

### supports

The event increases support but does not independently produce confirmation.

### confirms

The event establishes the item within the scenario.

### weakens

The event reduces support without fully excluding the item.

### excludes

The event supports movement to `ruled_out`.

### supersedes

The event displaces an earlier formulation.

### initiates_process

The event initiates or materially advances an investigation or workflow.

### resolves_process

The event completes the annotated investigation, referral, or workflow.

### insufficient

The event is related to the item but does not justify a state transition.

### irrelevant

The event has no material role for the item.

## Transition path

`gold_transition_path` records the state of each item after every evidence event.

Example:

This allows the scorer to identify:

* correct intermediate states;
* premature updates;
* delayed updates;
* missing updates;
* incorrect transition labels;
* first deviation from gold;
* recovery after an earlier error.

## Final transition

`gold_final_transition` describes the net transition from the prior state to the final state.

It is not always identical to the transition label at the final evidence event.

For example:

The path-level final event may be `reactivate`, while the net prior-to-final transition remains `confirm`.

The scorer evaluates both representations separately.

## Item evidence trace

`item_evidence_trace` lists all evidence events materially relevant to an item.

Example:

An event may belong to an item’s evidence trace without being decisive.

## Decisive evidence

`decisive_evidence` identifies the event or events responsible for the key state transition.

Example:

An empty list means no evidence event produced a decisive transition for that item.

This commonly occurs when the item is correctly preserved.

## Minimal decisive subsets

`minimal_decisive_subsets` identifies the smallest evidence set sufficient for the annotated final state or transition.

Example:

The empty subset represents a preserved item for which no evidence-driven state transition was required.

The schema supports more than one valid minimal subset, although most v0.1 records contain one annotated subset per item.

## Changed and preserved items

`changed_items` lists tracked items whose final state differs from their prior state.

`preserved_items` lists tracked items whose final state remains unchanged.

These fields allow direct measurement of:

* over-updating;
* under-updating;
* selective preservation;
* cross-item propagation errors.

## Integration types

The release uses the following high-level integration labels:

These labels support stratified analysis and are not model inputs in the standard benchmark task.

## Reasoning patterns

The release includes reasoning-pattern annotations such as:

These fields describe the structural reasoning mechanism exercised by a scenario.

They should remain hidden during ordinary test inference.

## Mechanism complexity

`mechanism_complexity` is an ordinal release-specific annotation.

The v0.1 values range from:

The field describes the relative structural complexity of the evidence-integration mechanism within this release.

It is not:

* a universal clinical difficulty score;
* a measure of patient severity;
* a validated cognitive-complexity scale.

The distribution is concentrated around levels 3 and 4. Broader complexity levels are planned for v0.2.

## Annotation confidence

All v0.1 records use:

This reflects the deliberately controlled synthetic design of the gold cases.

Future versions may introduce medium- and low-confidence scenarios where more than one interpretation remains defensible.

## Evaluation

Run the scorer with:

Write a complete JSON report with:

Use strict structural checking with:

## Primary metrics

The scorer reports:

## Structural diagnostics

The scorer also reports:

## Complete reconstruction accuracy

A scenario counts as a complete reconstruction only when the model correctly predicts:

This is the strictest metric in the benchmark.

A correct endpoint reached through an incorrect path does not count as a complete reconstruction.

## Pair-level evaluation

The scorer reports whether both members of each matched pair are correct.

Pair-level metrics include:

This prevents a model from receiving full pair credit for solving only the easier member.

## Order-sensitive evaluation

The `CMESI-001-A/B` training pair demonstrates evidence-order sensitivity.

The pair contains substantively equivalent evidence in different orders.

A successful model should:

* produce different intermediate paths where required;
* identify the decisive event at the correct step;
* converge to the same final state;
* preserve final-state invariance without erasing path differences.

## Recommended baseline experiments

The initial release supports four useful baselines.

### Final-state-only baseline

Predict only the final state of each tracked item.

This establishes how much performance is obtainable without trajectory reconstruction.

It is not eligible for complete-reconstruction scoring.

### Structured zero-shot baseline

Use the complete prediction schema with no examples.

Recommended decoding:

### Structured few-shot baseline

Use two to four training scenarios as demonstrations.

Demonstration scenario identifiers and exact prompt construction should be reported.

### Supervised fine-tuned baseline

Convert `data/train.jsonl` into model input-output pairs and fine-tune a model on the structured task.

Because the training split contains only 32 scenarios, results should be treated as pilot evidence.

At least three random seeds should be reported.

## Leakage controls

Do not:

* expose test gold fields to the model;
* use test scenarios as demonstrations;
* split matched pairs;
* tune prompts after inspecting item-level test errors;
* manually repair semantic model errors;
* use gold information during post-processing.

Permitted deterministic syntax repair may include:

* removing an outer Markdown fence;
* extracting one complete JSON object;
* normalising whitespace.

It must not include:

* changing ontology labels;
* adding missing evidence identifiers;
* repairing state-transition logic;
* filling fields using gold structure.

## Reporting requirements

Every reported experiment should include:

With only 16 held-out test scenarios, metric differences should be treated as exploratory unless they are large and consistent across seeds and structural measures.

## Intended uses

Appropriate uses include:

* structured reasoning evaluation;
* evidence-integration research;
* state-tracking research;
* prompt comparison;
* small supervised fine-tuning experiments;
* matched-pair analysis;
* trajectory-scoring research;
* model error analysis;
* benchmark architecture development.

## Limitations

### Synthetic data

All scenarios are synthetic and intentionally simplified.

They do not reproduce the full ambiguity, incompleteness, noise, documentation style, or operational complexity of real clinical records.

### Small release

The dataset contains 48 scenarios.

It is sufficient for controlled pilot evaluation but too small to support broad claims of learned clinical competence.

### Limited state machines

Investigation states are primarily represented through:

Treatment states are primarily represented through:

Richer workflow states are planned for v0.2.

### Moderate mechanism complexity

Most v0.1 examples have mechanism complexity 3 or 4.

The release contains relatively few long contradictory sequences, multi-treatment interactions, or high-complexity state cascades.

### Single gold trajectory

Most scenarios provide one canonical gold path.

The release does not yet support several equally valid clinical interpretations.

### No clinical deployment claim

Performance on CMESI does not demonstrate:

* medical competence;
* diagnostic reliability;
* treatment safety;
* regulatory compliance;
* readiness for clinical deployment.

## Planned v0.2 extensions

The next release is expected to add:

* mechanism-complexity levels 5 to 7;
* longer contradictory evidence sequences;
* evidence-quality reasoning;
* three-way competing diagnoses;
* cross-item dependency cascades;
* richer investigation states;
* richer treatment states;
* longer non-monotonic trajectories;
* historical evidence reinterpretation;
* investigation cancellation and repetition;
* treatment pause, resume, switch, escalation, and completion;
* confidence-stratified annotations;
* valid alternative gold paths.

The v0.1 release will remain frozen as the foundational benchmark layer.

## Generalisation beyond medicine

The core benchmark architecture is domain-independent:

The same structure could later support benchmarks in:

* legal case management;
* software debugging;
* cybersecurity incidents;
* industrial control;
* supply-chain management;
* scientific hypothesis revision;
* governance processes.

CMESI is the first clinical implementation of that broader state-transition benchmark architecture.

## Licence

The dataset is released under the Creative Commons Attribution 4.0 International licence.

See `LICENSE` for the full licence text.

## Citation

Preferred citation:

BibTeX: