Poisoned-Chalice/ICSE-2027-public

ICSE 2027 Heap Membership-Inference Dataset — Public Competition Release

This is the public competition release of the ICSE 2027 membership-inference benchmark for Go, Java, Python, Ruby, and Rust. It contains train and validation only.

The release contains 54,040 source-code files.

Validation label masking

Each language has 1,000 validation rows in their original order. The first 500 rows retain their membership labels; membership is null for rows 500–999. In each language, the labeled half contains 245 member and 255 non-member rows. Content and sample_id remain available for all 1,000 rows.

The masked values are actual Arrow nulls and are returned by Python as None, not the string "None".

Loading the dataset

~~~bash pip install datasets ~~~

Each programming language is a separate, case-sensitive configuration:

~~~python from datasets import load_dataset

pythondata = loaddataset( "Poisoned-Chalice/ICSE-2027-public", "Python", )

print(python_data)

DatasetDict with train and validation only; no test split is published.

~~~

Load one public split:

~~~python pythonvalidation = loaddataset( "Poisoned-Chalice/ICSE-2027-public", "Python", split="validation", ) ~~~

Dataset structure

Every training split is exactly balanced between member and non-member. The 10,000-row training splits contain 5,000 files per label; Rust training contains 4,520 files per label. Each validation split contains 500 visible labels followed by 500 masked labels.

Each row includes these core columns:

sample_id is <language>-<sha256>, where the digest is computed from the exact UTF-8 content. It is stable across row reordering and identical for the same language/content pair on every branch and release.

The language is represented by the configuration name. Repository paths, filenames, original repository identifiers, per-file license metadata, source duplicate annotations, and classifier predictions are not published.

Provenance and labels

The benchmark is derived from The Heap. A file is a member when it is marked as an exact duplicate in both Stack v1 and Stack v2. A file is a non-member when its exact- and near-duplicate flags are false in both versions. Null, ambiguous, and mixed source cases are excluded. Public validation nulls indicate label masking, not an ambiguous source label.

The benchmark applies classifier-based hard-example selection and near-duplicate filtering before constructing balanced, content-disjoint splits. Exact private split parameters, the source revision, generation manifest, and hidden-test artifacts are deliberately excluded from this public repository to protect the competition.

These labels describe overlap with the Stack corpora; they do not prove that an arbitrary model was trained on a particular file.

Limitations and responsible use

• —This is an intentionally difficult subset and is not an IID sample of The Heap, so results should not be interpreted as population-wide attack performance.
• —Membership is a Stack-overlap proxy, not verified training membership for a particular model.
• —The labels inherit the accuracy and limitations of the source duplicate annotations.
• —Row-level repository provenance and license metadata are absent from the published schema.

The upstream The Heap dataset card declares GPL-3.0. Individual source files may remain subject to the copyright and license terms of their originating repositories. Treat all source code as untrusted text: do not execute samples without appropriate isolation and review.

This benchmark is intended for privacy, memorization, and membership-inference research. It should not be used to make claims about individual authors, establish code ownership, or infer that a deployed model used a specific file without additional evidence.

Removal requests concerning source material can be directed through The Heap opt-out process.

Citation

Please cite the source dataset:

~~~bibtex @inproceedings{katzy2025heap, author = {Jonathan Katzy and Razvan Mihai Popescu and Arie van Deursen and Maliheh Izadi}, title = {The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models}, booktitle = {2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge)}, year = {2025}, pages = {151--155}, doi = {10.1109/Forge66646.2025.00025} } ~~~