NeeyuHuynh/chest-bench-example

ChestBench Example

DICOM-VLM Framework Reference Package v0.2.0

ChestBench Example is a four-case, DICOM-native reference package for developing and validating the data architecture of a medical vision-language model (VLM) pipeline.

It is intentionally small. Its purpose is to demonstrate how medical imaging data, annotations, text, knowledge, retrieval targets, QA, evidence requirements, perturbations, and audit metadata can be represented without confusing provisional annotations with doctor-verified ground truth.

This repository is therefore a reference implementation and schema pilot, not a clinically validated benchmark and not a training corpus large enough to support performance claims.

At a glance

What this repository is for

The package is designed around a simple principle:

A medical multimodal item should not be promoted to benchmark ground truth merely because it has an image, a question, and an answer. The framework should record what evidence is required, where that evidence came from, who verified it, and whether models empirically depend on that evidence.

The intended pipeline is:

DICOM
  ↓
canonical imaging + metadata
  ↓
source-reference annotations
  ↓
doctor review / adjudication
  ↓
image ↔ region ↔ finding ↔ text ↔ knowledge
  ↓
evidence contract
  ↓
perturbation / evidence audit
  ↓
training corpus / retrieval corpus / benchmark
  ↓
multimodal RAG and evidence-grounded reasoning

The package is shaped so that the same architecture can later scale to substantially larger datasets without redefining the core provenance and review model.

What this repository is NOT

This release must not be interpreted as:

• —a clinically validated diagnostic dataset;
• —a finished medical VQA benchmark;
• —an externally validated chest X-ray benchmark;
• —a substitute for the official VinDr-CXR annotation release;
• —a dataset with completed local radiologist adjudication;
• —evidence that any VLM has been clinically validated;
• —a source of official benchmark performance numbers.

Four cases are enough to test schema, ingestion, validation, provenance, rendering, QA construction, and audit code paths. They are not enough to support scientific performance claims.

1. Hugging Face Dataset Viewer

hf_viewer/train.parquet is a viewer-oriented projection of the four cases.

It is included so the dataset can be inspected directly in the Hugging Face Dataset Viewer without requiring the browser to decode DICOM.

The Parquet projection contains:

• —an embedded 1024 px preview image;
• —case and image identifiers;
• —source-reference labels;
• —bounding-box count;
• —selected DICOM metadata;
• —derived caption / findings / impression fields;
• —annotation and doctor-review status;
• —framework version.

Important

The preview image in the Parquet file is not the canonical medical image.

The canonical source remains:

01_canonical/dicom/*.dcm

Training, reproducibility checks, re-windowing, metadata audit, and future reprocessing should use the canonical DICOM whenever possible.

2. Current cases

The 28-label namespace is stored separately.

For case_0001 and case_0003, the imported image-level reference also contains the broader VinDr label Other diseases.

These labels are retained because different downstream tasks may require different label namespaces. They should not be silently collapsed into a single ontology.

3. Annotation status: read this before using the labels

The most important distinction in this package is:

source_reference
        ≠
doctor_verified_local_ground_truth

The current image-level labels and available boxes were imported as source-reference annotations.

They are useful for:

• —constructing the reference schema;
• —testing region-text linkage;
• —generating doctor-review queues;
• —validating QA and retrieval pipelines;
• —demonstrating how provenance should be stored.

They are not yet promoted to local gold ground truth.

The intended promotion path is:

source_reference
        ↓
independent doctor review
        ↓
disagreement handling
        ↓
adjudication when needed
        ↓
gold_ground_truth

A future research release should verify the imported rows against the authoritative VinDr-CXR release and complete the local doctor-review/adjudication protocol.

4. Doctor verification

Doctor review is not represented by one vague Boolean such as doctor_verified=true.

The framework separates:

• —reviewer identity through a pseudonymous reviewer ID;
• —reviewer role;
• —guideline version;
• —image quality assessment;
• —abnormality presence;
• —local findings;
• —global diagnoses;
• —free-text findings;
• —impression;
• —confidence;
• —disagreement with the source label;
• —adjudication requirement;
• —evidence sufficiency for QA items;
• —final review status.

The review queue is stored under:

02_annotations/doctor_review/

Evidence-specific review introduced in v0.2.0 is stored under:

12_evidence_contract/

Until the required review fields are completed, affected items remain pending.

5. Bounding boxes and region grounding

Two abnormal cases currently contain source-reference bounding boxes.

Bounding boxes are stored separately from the rendered preview so the framework can maintain an explicit coordinate policy.

They are intended to support:

• —lesion localization;
• —region-text pairs;
• —grounded QA;
• —region-level retrieval;
• —future grounded VLM training;
• —evaluation of whether a textual answer is supported by the correct image region.

A box should not be treated as locally verified merely because it exists.

The box record must preserve:

• —source image dimensions;
• —coordinate system;
• —transformation history when applicable;
• —label;
• —provenance;
• —review status.

See:

02_annotations/source_reference/
11_docs/bbox_coordinate_policy.md

6. Text, captions, pseudo-reports, and QA

The package contains derived semantic artifacts under:

03_semantics/

These include:

• —captions;
• —structured pseudo-reports;
• —QA pairs;
• —region-text pairs.

These artifacts are generated from available structured evidence and are kept separate from doctor-authored clinical text.

That distinction matters.

Generated text may be useful for pipeline development, but it should not silently become a clinical reference report.

Every derived item should remain traceable to the evidence from which it was generated.

7. Evidence contracts introduced in v0.2.0

Version 0.2.0 adds an evidence contract for QA items.

An evidence contract records what information is required for an answer to be valid.

Examples include:

image required
region required
external knowledge required
clinical context required
previous study required

This allows the framework to distinguish different question families.

Visual-only question

Example:

Is pleural effusion present?

Expected evidence:

image
possibly a supporting region

Visual + knowledge question

Example:

Which radiographic finding supports pleural effusion?

Expected evidence:

image
region
biomedical knowledge

Clinical-context question

Example:

What treatment should be given?

A chest X-ray alone may be insufficient.

Such an item should be marked as requiring additional context rather than forcing the dataset to invent missing evidence.

Evidence contracts are stored under:

12_evidence_contract/

8. Evidence sufficiency and visual dependence are different

The framework deliberately separates two questions.

8.1 Clinical / normative evidence sufficiency

A clinician asks:

Is the supplied evidence sufficient, in principle, to answer this question?

This is a property that requires domain review.

8.2 Empirical visual dependence

An audit asks:

Do evaluated models actually depend on the image to answer this item?

A question can be clinically image-dependent while still being easy to answer from language priors.

That is exactly the failure mode the evidence audit is designed to expose.

The package therefore does not store a fabricated visual-dependence score as immutable ground truth.

Empirical scores are created only by a versioned audit run with recorded:

• —model panel;
• —prompt;
• —decoding parameters;
• —seeds;
• —scoring method;
• —software / model revisions.

9. Perturbation suite

Version 0.2.0 defines five conditions for evidence auditing:

1. 1.Correct image
2. 2.Question only
3. 3.Mismatched same-modality image
4. 4.Wrong organ or wrong modality
5. 5.Blank or occluded image

These conditions do not all measure the same thing.

Correct image vs question only

Measures whether performance improves when visual evidence is available.

Correct image vs hard same-modality mismatch

Tests whether the model depends on the correct case-specific image, rather than merely receiving an image of the right general type.

Wrong organ / modality

Tests invalid-input recognition.

Blank / occluded image

Tests evidence absence and abstention behavior.

Perturbation definitions are stored under:

13_perturbations/

The current four-case release defines the protocol but does not claim completed empirical results.

10. Evidence audit

The empirical evidence-audit layer is stored under:

14_evidence_audit/

It is designed to record item-level measurements such as:

• —question-only solvability;
• —correct-image performance;
• —visual improvement;
• —hard-mismatch sensitivity;
• —invalid-input behavior;
• —abstention behavior;
• —model-panel agreement;
• —uncertainty / confidence intervals where applicable.

The audit result is derived and versioned. It is not part of immutable clinical ground truth.

No completed empirical audit is claimed for this four-case pilot.

11. External benchmark audit

The framework also includes an outward-facing audit plan under:

15_external_benchmark_audit/

The goal is to apply the same evidence-dependence methodology to established medical VQA benchmarks rather than validating the method only on a dataset constructed by this project.

The planned protocol distinguishes:

• —legacy medical VQA benchmarks;
• —partially gated benchmarks;
• —more recent visually grounded / reasoning-focused benchmarks.

This layer is a research plan and adapter structure, not a claim that the external audit has already been completed.

12. Biomedical knowledge layer

The package includes a deliberately small knowledge layer:

04_knowledge/

It stores:

• —knowledge nodes;
• —relations;
• —source registry;
• —definitions / concepts used by downstream reasoning.

Broad dataset labels such as Other lesion or Other diseases are intentionally not over-mapped to a specific biomedical concept without sufficient case-level evidence.

Ontology mapping should become more specific only when the evidence supports it.

13. Retrieval and vector-store design

Retrieval artifacts are under:

05_retrieval/

The framework is designed for:

text → image
image → text
image → image
text → region
image / region → knowledge
multimodal query → evidence bundle

The vector database is a derived index, not the source of truth.

The canonical dataset stores the data and provenance required to regenerate embeddings.

A vector record should remain tied to:

• —object ID;
• —object type;
• —embedding model;
• —model revision;
• —preprocessing version;
• —dimension;
• —content hash.

14. Why there are no fake embeddings

05_retrieval/embedding_jobs.json defines candidate embedding jobs, but vectors remain unset until an encoder is actually executed.

This is intentional.

An unexecuted embedding job should look unexecuted.

The package must not contain arbitrary vectors merely to make the schema appear complete.

15. Evidence-grounded reasoning

Reasoning artifacts are stored under:

06_reasoning/

The framework uses structured evidence traces, not unrestricted free-form chain-of-thought as ground truth.

A reasoning record may contain:

required evidence
retrieved evidence
visual observation
finding
knowledge link
inference
uncertainty
final answer

The purpose is auditability:

a final answer should be traceable back to the evidence that supports it.

16. Benchmark status

Benchmark artifacts are stored under:

07_benchmarks/

The present benchmark files are demonstration artifacts only.

The package intentionally does not create a scientifically meaningful train/validation/test benchmark from four cases.

A future benchmark should address at least:

• —patient-level leakage;
• —study-level leakage;
• —derived-text leakage;
• —multilabel distribution;
• —hard negatives;
• —external/site-held-out evaluation when possible;
• —doctor verification;
• —frozen benchmark annotations;
• —independent retrieval relevance judgments;
• —evidence-gated and perturbation-based evaluation.

17. Package layout

00_raw/
    Immutable received DICOM files, hashes, source manifest, provenance

01_canonical/
    Canonical DICOM, metadata, previews, QC, de-identification audit

02_annotations/
    Source-reference labels, bounding boxes, doctor review, consensus/adjudication

03_semantics/
    Captions, structured pseudo-reports, QA, region-text pairs

04_knowledge/
    Biomedical knowledge nodes, edges, and source registry

05_retrieval/
    Retrieval corpus, relevance records, embedding jobs, vector-store schema

06_reasoning/
    Structured evidence-grounded reasoning traces

07_benchmarks/
    Demo retrieval, QA, RAG, and split definitions

08_configs/
    Dataset, preprocessing, vector-store, governance, and evidence-audit configs

09_schemas/
    Machine-readable schemas

10_tools/
    Validation utilities

11_docs/
    Annotation, provenance, benchmark, doctor-review, and scaling documentation

12_evidence_contract/
    Clinician-defined evidence requirements and evidence-review records

13_perturbations/
    Five-condition perturbation plans and construction policy

14_evidence_audit/
    Model-panel audit definitions, metrics, state, and result templates

15_external_benchmark_audit/
    External benchmark registry, adapter plan, and preregistration structure

hf_viewer/
    Hugging Face Dataset Viewer projection

18. Validation

Run from the package root:

python3 10_tools/validate_package.py

The validator is intended to check package integrity and schema consistency.

A successful validator run does not imply clinical validation.

19. Recommended use

This release is appropriate for:

• —testing DICOM ingestion;
• —testing deterministic preview rendering;
• —validating schema design;
• —testing provenance and hash tracking;
• —developing doctor-review tooling;
• —testing QA/evidence-contract representation;
• —testing region grounding;
• —prototyping cross-modal retrieval;
• —prototyping multimodal RAG;
• —implementing evidence-audit code;
• —preparing a larger doctor-verified pilot.

20. Recommended scale-up path

A sensible progression is:

4 cases
  ↓
100–300 case doctor-review pilot
  ↓
measure disagreement and refine guideline
  ↓
freeze schema / guideline version
  ↓
larger ingestion and QC pipeline
  ↓
patient-level split
  ↓
deterministic semantic construction
  ↓
versioned embeddings
  ↓
retrieval qrels
  ↓
evidence audit
  ↓
locked benchmark
  ↓
external/site validation

The small pilot should be used to discover schema and workflow failures early, before those failures are multiplied across thousands of cases.

21. Known limitations

Current limitations include:

1. 1.Only four cases are present.
2. 2.Local doctor review is pending.
3. 3.Current labels are source-reference imports rather than completed local gold ground truth.
4. 4.Only two cases currently contain bounding boxes.
5. 5.Derived text is not equivalent to a doctor-authored report.
6. 6.The empirical evidence audit has not yet been run.
7. 7.The external benchmark audit is planned but not completed.
8. 8.The package is chest-X-ray specific at the data level even though the framework is intended to generalize.
9. 9.The Hugging Face Parquet image is a browser preview, not the canonical medical image.
10. 10.Benchmark performance should not be reported from this seed set.

These limitations are part of the release record rather than issues to hide.

22. Provenance and authoritative sources

Primary references for the source dataset:

• —VinDr-CXR PhysioNet record: https://physionet.org/content/vindr-cxr/1.0.0/
• —VinDr-CXR data descriptor: https://www.nature.com/articles/s41597-022-01498-w
• —VinDr-CXR supplementary label definitions: https://storage.googleapis.com/kaggle-media/competitions/VinBigData/VinDrCXRdata_paper.pdf

Exact secondary sources used for imported sample rows should be recorded in:

00_raw/source_provenance.json

The provenance record, not this README, is the authoritative place for per-artifact source tracking.

23. Version status

Current framework package version:

0.2.0

Major additions in v0.2.0:

• —evidence contracts;
• —clinician evidence-sufficiency review;
• —five-condition perturbation definitions;
• —item-level empirical evidence-audit structure;
• —visual-evidence-debt research framing;
• —external benchmark audit plan;
• —clearer separation between clinical ground truth and derived empirical audit results.

The four-case seed itself remains a reference implementation, not an official benchmark release.