Krows7/CraftSight-Minecraft

CraftSight

<p align="center"> <img src="https://img.shields.io/badge/version-1.0.1-blue" alt="Version"> </p>

Frame-level multi-label visual annotations for Minecraft agents.

Dataset creation toolkit: CraftSight is built and maintained with CraftSight Labeler, an open-source, browser-based annotation tool and reproducible release pipeline for multi-label Minecraft vision datasets. It supports manual and model-assisted labeling, structured game-state annotations, and trajectory-safe train/validation/test splits.

CraftSight provides Minecraft gameplay frames annotated with terrain, hazards, blocks, interfaces, structures, mobs, and scene-level context. It is intended for multi-label perception, representation learning, active learning, and embodied-agent research.

Not an official Minecraft product. Not approved by or associated with Mojang or Microsoft.

Minecraft is a trademark of the Microsoft group of companies. Minecraft-related rights remain with Mojang, Microsoft, and their licensors.

Dataset overview

CraftSight exposes three configurations:

core and full contain exactly the same active frames and use the same trajectory-level train/validation/test assignment. They differ only in their label_* columns.

Quick start

from datasets import load_dataset

dataset = load_dataset("Krows7/CraftSight-Minecraft", "core")

print(dataset)

example = dataset["train"][0]
image = example["image"]

print(image)
print(image.size)
print(example["label_water"])

The image column is created automatically by Hugging Face ImageFolder. Accessing example["image"] returns a decoded Pillow image; no manual path joining, download call, or Image.open() step is required.

core is the default configuration:

dataset = load_dataset("Krows7/CraftSight-Minecraft")

Load all 48 labels:

full = load_dataset("Krows7/CraftSight-Minecraft", "full")

Load the unlabeled image index:

unlabeled = load_dataset(
    "Krows7/CraftSight-Minecraft",
    "unlabeled",
    split="train",
)

example = unlabeled[0]
image = example["image"]

Working with labels

Every target is stored in a separate label_<name> column.

Never treat `-1` as a negative label.

label_columns = [
    column
    for column in dataset["train"].column_names
    if column.startswith("label_")
]

Example masked binary cross-entropy:

import torch
import torch.nn.functional as F

# logits and targets have shape [batch_size, number_of_labels]
known = targets != -1
safe_targets = targets.clamp_min(0).float()

loss_per_target = F.binary_cross_entropy_with_logits(
    logits,
    safe_targets,
    reduction="none",
)

loss = loss_per_target[known].mean()

The canonical label order and numeric semantics are defined in `label_schema.json`. Class boundaries and edge cases are documented in `label_definitions.md`.

Images

The Viewer-facing CSV files store image references in a file_name column:

images/<namespace>/<relative-path>.webp

Access decoded images

from datasets import load_dataset

dataset = load_dataset(
    "Krows7/CraftSight-Minecraft",
    "core",
    split="train",
)

example = dataset[0]
image = example["image"]

print(type(image))
print(image.mode)
print(image.size)

Access the underlying path without decoding

from datasets import Image, load_dataset

dataset = load_dataset(
    "Krows7/CraftSight-Minecraft",
    "core",
    split="train",
)

paths = dataset.cast_column(
    "image",
    Image(decode=False),
)

print(paths[0]["image"])

Depending on the loading mode, this returns a local cached path, a remote path, or embedded bytes.

Stream examples without materializing the full dataset

from datasets import load_dataset

stream = load_dataset(
    "Krows7/CraftSight-Minecraft",
    "core",
    split="train",
    streaming=True,
)

example = next(iter(stream))
image = example["image"]

Image provenance and permitted uses differ by data_source; review `THIRD_PARTY_NOTICES.md` before downloading, redistributing, or using images commercially.

Choosing a configuration

core

Use core for:

• —headline benchmark results;
• —model comparison;
• —standard multi-label training;
• —threshold selection on validation data;
• —experiments requiring positive support for every target in every split.

Core labels:

water
drop_off
dark_cave
sand_or_redsand
tree_log
leaves
stone
inventory_open
hostile_near
wood_planks
cobblestone
torch
water_bucket_in_hand
building
hostile_present
sky_visible
open_surface
mountains
glass_pane_or_window

full

Use full for:

• —rare-label and long-tail studies;
• —few-shot experiments;
• —active learning;
• —label completion;
• —schema extension;
• —partially labeled learning.

The full configuration preserves all 48 schema labels. Some classes have limited support, and furnace and ladder currently have no positive active examples.

<details> <summary><strong>Show all 48 labels</strong></summary>

water
lava
fire
drop_off
dark_cave
web_cobweb
ice
sand_or_redsand
gravel
tree_log
leaves
stone
crafting_table
furnace
chest
bed
inventory_open
underwater
hostile_near
wood_planks
coal_ore
iron_ore
cobblestone
door
ladder
torch
water_bucket_in_hand
building
player_damage
hostile_present
creeper
zombie
skeleton
spider
enderman
witch
cow
sheep
pig
chicken
villager
sky_visible
open_surface
farmland
crop
ocean
mountains
glass_pane_or_window

</details>

unlabeled

Use unlabeled for:

• —self-supervised pretraining;
• —image feature extraction;
• —embedding generation;
• —clustering and similarity search;
• —trajectory representation learning;
• —active-learning candidate selection.

The loaded configuration contains an image feature and trajectory metadata:

image
data_source
agent
task
seed
episode
frame_index
trajectory_id

Splits

The supervised configurations use a leakage-resistant group split:

All frames from one trajectory_id remain in one split. No trajectory appears in more than one split.

The canonical assignment is stored in:

splits/core/splits_by_trajectory.csv

The full configuration reuses the same manifest. Do not create a new random frame-level split, because adjacent frames may be nearly identical.

Dataset structure

Repository layout

.
├── README.md
├── LICENSE
├── LICENSE_SCOPE.md
├── THIRD_PARTY_NOTICES.md
├── label_schema.json
├── state_schema.json
├── label_definitions.md
├── classes_core.txt
├── classes_full.txt
├── metadata.csv
├── all_images.txt
├── core_train.csv
├── core_val.csv
├── core_test.csv
├── full_train.csv
├── full_val.csv
├── full_test.csv
├── unlabeled.csv
└── images/
    ├── basalt/
    ├── mobs/
    └── custom/

metadata.csv is the canonical annotation table and includes both active and ignored rows.

The root core_*.csv, full_*.csv, and unlabeled.csv files are Viewer-ready ImageFolder metadata tables. Their file_name values point into the shared images/ directory and are exposed by datasets as the image feature.

all_images.txt is the canonical list of all 64,279 image paths. unlabeled.csv contains exactly the 61,221 paths absent from metadata.csv.

Annotated row fields

Identity and trajectory metadata

Annotation metadata

Optional game state

The state schema is defined in `state_schema.json`. State is a separate modality and must not silently replace visual evidence.

Supported tasks

CraftSight supports:

• —multi-label image classification;
• —image feature extraction;
• —visual representation learning;
• —scene and hazard recognition;
• —perception modules for embodied and reinforcement-learning agents;
• —active-learning and partially labeled learning research.

CraftSight is not an object-detection dataset. It does not provide bounding boxes, instance masks, or object coordinates.

Evaluation

Use core for comparable benchmark results.

Recommended metrics:

• —macro F1;
• —micro F1;
• —mean average precision;
• —per-class average precision;
• —per-class precision and recall;
• —known-target and positive-example counts for each class.

Select thresholds only on validation data. Mask all -1 targets during training and evaluation.

Do not report a class metric for a split with no known positive examples for that class.

Dataset creation

Image sources

Image counts by namespace:

See `THIRD_PARTY_NOTICES.md` for source links, attribution requirements, and source-specific conditions.

Annotation process

Active annotations were produced through:

Annotations are image-level and multi-label. Multiple labels may be positive in one frame. Annotation rules prioritize visible evidence from the current frame rather than inference from adjacent frames, task names, or hidden game state.

Quality control

The tabular release was checked for:

• —schema and column-order consistency;
• —valid target values;
• —duplicate image paths;
• —path-derived metadata consistency;
• —state-field consistency;
• —split coverage;
• —trajectory leakage;
• —alignment between core and full;
• —positive-label support across core splits;
• —exact partitioning of annotated and unlabeled image paths.

The current release has:

• —no duplicate annotated paths;
• —no overlap between metadata.csv and unlabeled.csv;
• —complete coverage of all_images.txt;
• —no trajectory leakage;
• —exact agreement between core and full split assignments.

Limitations

• —Labels are highly imbalanced.
• —furnace and ladder have no positive active examples.
• —Several full labels are rare or absent from individual validation/test splits.
• —Only 50 annotated rows contain game-state metadata.
• —Frames within a trajectory remain temporally correlated.
• —Sources, tasks, and agents are not uniformly represented.
• —Some semantic label relationships are not mechanically enforced in every row.
• —frame_index is unavailable for some source filenames.
• —Performance may change with game version, edition, texture pack, shaders, field of view, resolution, and UI scale.

Users should report per-class support and inspect failure cases before drawing broad conclusions.

Licensing

CraftSight uses a layered licensing model.

Annotation and metadata layer

Maintainer-created annotations, normalized metadata, schemas, class lists, definitions, split manifests, and covered data organization are provided under CDLA-Sharing-1.0.

• —The unmodified agreement is in `LICENSE`.
• —The covered repository layer is defined in `LICENSE_SCOPE.md`.
• —Published modified or extended covered data must remain under the unmodified CDLA-Sharing-1.0.
• —Changed data files must be identified.
• —Existing attribution and practical source links must be preserved.

The CDLA does not impose restrictions on qualifying computational Results, subject to the agreement's definitions and terms.

Image layer

Images are not relicensed under CraftSight's CDLA grant:

• —mobs images are subject to CC BY-NC 4.0;
• —basalt images follow the terms stated by the upstream Zenodo record;
• —custom images remain subject to Minecraft and other applicable rights.

See `THIRD_PARTY_NOTICES.md` before redistribution or commercial use.

Citation

@dataset{craftsight_2026,
  author    = {Shaga, Konstantin},
  title     = {CraftSight: A Multi-label Perception Dataset for Minecraft Agents},
  year      = {2026},
  publisher = {Hugging Face},
  version   = {1.0.1},
  url       = {https://huggingface.co/datasets/Krows7/CraftSight-Minecraft}
}

Maintenance

Use the repository Discussions tab for annotation corrections, provenance updates, licensing questions, and schema proposals.