mehmetkeremturkcan/SAM3-Joint-Surgical-Dataset

SAM3 Joint Surgical Dataset

<p align="center"> <a href="https://keremturkcan.com/">Mehmet Kerem Turkcan</a>, Soham Samal, Zoran Kostic <br> <a href="https://www.aidl.ee.columbia.edu/">AIDL Lab, Columbia University</a>, <a href="https://www.i-pal.org/">IPAL Lab</a> </p>

<p align="center"> <a href="assets/surgicalsam3overview.png"><img src="assets/surgicalsam3overview.png" width="100%" alt="Four laparoscopic images from CholecInstanceSeg, CholecSeg8k, Endoscapes, and DSAD with human reference masks for instruments and anatomy"></a> </p>

The SAM3 Joint Surgical Dataset contains 41,372 physical images with 175,674 COCO annotations for surgical instruments, anatomy, and tissue. Its schema contains 29 categories and consolidates labels from CholecInstanceSeg, CholecSeg8k, the Dresden Surgical Anatomy Dataset, and Endoscapes. Positive annotations cover 25 categories. The test split contains positive instances for 24 categories.

<p align="center"> <a href="assets/resultssummary.png"><img src="assets/resultssummary.png" width="100%" alt="Matched grouped test results for the base SAM 3 checkpoint and the joint surgical model"></a> </p>

The matched evaluation uses the same images, text prompts, annotation scopes, postprocessing, and COCO evaluator for both checkpoints on the grouped test split of 4,134 images. Box AP and mask AP each report mean interpolated precision over 10 IoU thresholds, 101 recall points, and the 24 categories with positive test instances, with at most 100 detections per image.

<p align="center"> <a href="assets/instrumentexamples1.png"><img src="assets/instrumentexamples1.png" width="100%" alt="Human reference masks for grasper, hook, irrigator, and clipper"></a> </p>

<p align="center"> <a href="assets/instrumentexamples2.png"><img src="assets/instrumentexamples2.png" width="100%" alt="Human reference masks for bipolar, scissors, snare, and surgical instrument"></a> </p>

<p align="center"> <a href="assets/anatomytissueexamples1.png"><img src="assets/anatomytissueexamples1.png" width="100%" alt="Human reference masks for abdominal wall, liver, gastrointestinal tract, fat, connective tissue, and cystic duct"></a> </p>

<p align="center"> <a href="assets/anatomytissueexamples2.png"><img src="assets/anatomytissueexamples2.png" width="100%" alt="Human reference masks for gallbladder, hepatic vein, liver ligament, cystic plate, Calot triangle, and cystic artery"></a> </p>

<p align="center"> <a href="assets/anatomytissueexamples3.png"><img src="assets/anatomytissueexamples3.png" width="100%" alt="Human reference masks for colon, pancreas, small intestine, spleen, and stomach"></a> </p>

The galleries visualize human reference masks. Of the 25 examples, 24 use validation or test annotations. Hepatic vein uses a training annotation because it has no validation or test instances.

<p align="center"> <a href="assets/perconceptresults.png"><img src="assets/perconceptresults.png" width="100%" alt="Box and mask average precision for each category on the grouped test split"></a> </p>

The conversion pipeline merges records that resolve to the same physical frame before the split builder assigns each group identifier, which denotes a complete procedure, video, or patient, to one partition. Every pair of splits has zero overlap in group identifiers, physical frame components, normalized frame identifiers, and SHA-256 image digests.

Surgical SAM 3 was trained on this dataset.

Dataset summary

The split assignment targeted 80 percent training, 10 percent validation, and 10 percent test images while assigning every complete group to one split. Because group sizes differ, the resulting image counts approximate these targets.

Group composition

The training loader uses sampling_repeat to increase the contribution of source streams with fewer physical images. Summing sampling_repeat across the 33,094 training images yields 52,747 repeated training records per epoch. The distributed sampler pads this index to 52,752 samples across eight ranks. Validation and test evaluate each physical image once.

Physical frame merge

The merge begins with 48,267 source image records and 187,642 source annotations.

The pipeline joins records when their image bytes have the same SHA-256 digest or when their normalized source identifiers denote the same frame. The resulting identity graph contains 34,483 components with one record, 6,886 components with two records, and 3 components with four records.

Each component retains one canonical image file. The pipeline converts each compatible mask to compressed COCO RLE and attaches the retained instance annotations to one canonical image record. When a component contains CholecInstanceSeg and CholecSeg8k annotations, the merge retains the typed CholecInstanceSeg tool masks and their generic counterparts, then removes the coarser CholecSeg8k generic tool masks.

Grouped split

The split builder uses one group identifier for each normalized Cholec procedure, Endoscapes video, or Dresden patient. Before assignment, the builder treats ten normalized procedure identifiers as shared between CholecSeg8k and CholecT50.

The assignment uses seed 20260801 and a mixed integer linear program with the following constraints:

1. 1.Every complete group identifier belongs to exactly one split.
2. 2.Every source family contributes at least one group identifier to every split.
3. 3.A category supported by at least three group identifiers contributes at least one group identifier to every split.
4. 4.A category supported by two group identifiers contributes one to training and one to test.
5. 5.A category supported by one group identifier contributes that group identifier to training.
6. 6.Every Endoscapes group identifier whose retained supervision consists only of generated tool masks belongs to training.

The HiGHS solver reported an optimal assignment with a relative MIP gap of 0.0.

The split method and category support audit are provided in `docs/SPLIT_AUDIT.md`.

Categories

Each split cell gives the image count first and the annotation count second, in parentheses. A zero indicates that the split contains no positive annotation for that category.

Inferior mesenteric artery, intestinal veins, ureter, and vesicular glands are unsupported by positive supervision. They remain in the shared schema as placeholders for compatibility. Hepatic vein occurs in one training group. Test metrics cover the 24 categories with positive test instances.

Source annotation scopes

The exhaustive_category_ids field stores the union of the applicable source scopes for each merged image. Four schema categories without positive annotations are excluded from this field.

Visible structures can remain unlabeled when they fall outside a source scope. A missing annotation is therefore a negative label only for a category listed in exhaustive_category_ids.

CholecInstanceSeg

CholecInstanceSeg provides polygon masks for grasper, hook, irrigator, clipper, bipolar, scissors, and snare. Each typed instrument also receives the generic surgical instrument category.

CholecSeg8k

The converter maps the source semantic palette to abdominal wall, liver, gastrointestinal tract, fat, connective tissue, cystic duct, gallbladder, hepatic vein, and liver ligament. Instrument pixels map to surgical instrument. Masks with areas smaller than 400 pixels are excluded during source conversion.

Dresden Surgical Anatomy Dataset

The dataset contains frames from the subset with annotations for multiple anatomy categories. All retained anatomy masks for a physical frame are stored together. Masks with areas smaller than 400 pixels are excluded during source conversion.

Endoscapes

The anatomy stream provides gallbladder, cystic duct, cystic artery, cystic plate, and Calot triangle. Tool annotations map to surgical instrument.

The base SAM 3 interactive image model generated candidate tool masks from source bounding boxes for 820 training images that lack source masks. Candidates with predicted IoU below 0.5 were excluded. Validation and test contain zero images from this generated mask stream.

COCO format

Each split annotation file contains the standard COCO fields and additional image provenance fields:

images:
  id, file_name, height, width
  procedure_group, physical_component_id
  source_records, source_streams
  normalized_frame_ids, sha256_values
  exhaustive_category_ids, sampling_repeat

annotations:
  id, image_id, category_id
  bbox, area, iscrowd, segmentation

categories:
  id, name, supercategory

Bounding boxes use COCO [x, y, width, height] coordinates. Masks use compressed COCO RLE with [height, width] in segmentation.size. All annotations have iscrowd = 0.

sampling_repeat is training metadata. Evaluation should process every physical image once. exhaustive_category_ids is required for training and evaluation that respect annotation scope.

Repository layout

annotations/
  train.json
  validation.json
  test.json
data/
  train-*.tar
  validation-*.tar
  test-*.tar
manifest.jsonl
docs/
  group_assignment.json
  identity_audit.json
  split_audit.json
  SPLIT_AUDIT.md
pipeline/
  build_grouped_split.py
scripts/
  audit_splits.py
  package_webdataset.py
  render_concept_gallery.py
assets/
  surgical_sam3_overview.png
  instrument_examples_*.png
  anatomy_tissue_examples_*.png
  results_summary.png
  per_concept_results.png
  concept_gallery_manifest.json
README.md
LICENSE
THIRD_PARTY.md

manifest.jsonl maps each physical image to its TAR shard and member, and records the image hash, source lineage, annotation scope, and sampling repeat. The packaged validation split corresponds to the local builder name valid.

Loading the data

Download the dataset from Hugging Face:

hf download mehmetkeremturkcan/SAM3-Joint-Surgical-Dataset \
  --repo-type dataset \
  --revision main \
  --local-dir sam3_joint

Extract one split:

cd sam3_joint
mkdir -p train
for shard in data/train-*.tar; do
  tar -xf "$shard" -C train
done

Load one image and its annotations:

from pathlib import Path

from PIL import Image
from pycocotools.coco import COCO

root = Path("sam3_joint")
coco = COCO(root / "annotations" / "train.json")
image_id = coco.getImgIds()[0]
record = coco.loadImgs([image_id])[0]
image = Image.open(root / "train" / record["file_name"]).convert("RGB")
annotations = coco.loadAnns(coco.getAnnIds(imgIds=[image_id]))
evaluation_scope = record["exhaustive_category_ids"]
sampling_repeat = record["sampling_repeat"]

Limitations

The schema contains four categories without positive annotations. Hepatic vein has positive annotations from one training group and no validation or test support.

Several test categories have limited image support. Snare and spleen each appear in 2 test images, while small intestine appears in 4. Metrics for these categories have high sampling uncertainty.

The source datasets use different taxonomies and annotation protocols. exhaustive_category_ids records the categories for which each image supplies valid presence and absence supervision.

The Endoscapes generated masks provide training supervision and are identified by the source provenance fields. Validation and test use masks supplied by the source datasets.

The images represent specific laparoscopic procedures, institutions, and acquisition systems. The dataset and associated model have not been validated for clinical decision making or patient care.

Licenses and citations

Source images and annotations retain their original licenses and citation requirements. CC BY-NC-SA 4.0 applies only to the original compilation metadata and documentation.

`THIRD_PARTY.md` records source citations, licenses, and compilation changes. Several source licenses restrict commercial use.

Citation

@misc{turkcan2026sam3jointsurgicaldataset,
  title        = {SAM3 Joint Surgical Dataset},
  author       = {Turkcan, Mehmet Kerem and Samal, Soham and Kostic, Zoran},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/mehmetkeremturkcan/SAM3-Joint-Surgical-Dataset}}
}

Users must also cite the annotation sources and their underlying image datasets. Complete bibliographic records are provided in `THIRD_PARTY.md`.

Authors

Mehmet Kerem Turkcan, Soham Samal, and Zoran Kostic. This project was developed by the AIDL Lab at Columbia University and IPAL Lab.