hotchpotch/cornstack-samples

cornstack-samples

🚧 This dataset is under active development and may change.

Filtered CoRNStack sample subsets for code retrieval training.

Source dataset and paper:

• —CoRNStack collection: https://huggingface.co/collections/nomic-ai/cornstack
• —CoRNStack paper: https://huggingface.co/papers/2412.01007

Note: the original CoRNStack collection is a much larger dataset family for code search training. If you need large-scale data (not samples), please refer to the original CoRNStack collection above.

What This Release Contains

This release keeps the original subset layout (6 languages x pair + hard-negatives) and applies deterministic rule-based filtering.

In this revision, query-level deduplication and positive-document (pos) deduplication are applied per subset. For both rules, if duplicates exist, only the first row is kept.

Config Layout And Schema

Each language is published as two configs with split train:

• —{lang}-v1-pair-2M
• —{lang}-v1-hard-negatives-100k

Schema:

• —Pair configs: query, pos
• —Hard-negative configs: query, pos, negs (list[string])

Subsets And Row Counts (After Filter + Query Dedup + Pos Dedup)

Note: subset names keep the original 2M / 100k naming convention for compatibility, but actual row counts vary by language and can be substantially lower after quality filtering and deduplication.

Total rows:

• —Pair: 8,524,322
• —Hard-negatives: 487,478
• —Overall: 9,011,800

Dedup Impact (Pos Dedup Stage)

The table below shows the pos-dedup impact on top of the previous filtered + query-deduped data.

Stage totals:

• —Pair removed by pos dedup: 145,550 (1.68%)
• —Hard-negatives removed by pos dedup: 2,671 (0.54%)
• —Overall removed by pos dedup: 148,221 (1.62%)

Quick Usage

from datasets import load_dataset

pair_ds = load_dataset("hotchpotch/cornstack-samples", "python-v1-pair-2M", split="train")
hard_ds = load_dataset("hotchpotch/cornstack-samples", "python-v1-hard-negatives-100k", split="train")

print(pair_ds.column_names, len(pair_ds))
print(hard_ds.column_names, len(hard_ds))

License

This dataset follows CoRNStack and is released under Apache-2.0.

Citation And Attribution

If you use this dataset, please cite and attribute CoRNStack:

• —Paper: https://huggingface.co/papers/2412.01007
• —Collection: https://huggingface.co/collections/nomic-ai/cornstack

Noise Filtering Algorithm (Rule-based)

The following deterministic rules are applied before publishing this release.

1. 1.Prefix-based noisy query removal A row is dropped if query starts with any of the following prefixes:
2. 2.TODO
3. 3.GET /
4. 4.POST /
5. 5.PUT /
6. 6.DELETE /
7. 7.Display a listing of the resource.
8. 8.Store a newly created resource in storage.
9. 9.Show the form for editing the specified resource.
10. 10.Update the specified resource in storage.
11. 11.Show the form for creating a new resource.
12. 12.Remove the specified resource from storage.
13. 13.Display the specified resource.
14. 14.Transform the resource into an array.
15. 15.Autogenerated method stub
16. 16.Auto generated
17. 17.this down() migration is autogenerated
18. 18.this up() migration is autogenerated
19. 19."/ renamed from:"
20. 20."/ access modifiers changed from:"

1. 1.Minimum positive-document length A row is dropped if the positive side text is shorter than 30 characters.
2. 2.Pair configs: pos length >= 30 required
3. 3.Hard-negative configs: pos length >= 30 required

1. 1.Hard-negative size constraint For hard-negative configs in this release, each row keeps exactly seven negatives after normalization (len(negs) = 7, equivalent to min_negs = 7).

1. 1.Query-level deduplication Within each subset split, rows are grouped by exact query string.
2. 2.Keep the first occurrence
3. 3.Drop all later duplicates

1. 1.Positive-document (pos) deduplication Within each subset split, rows are grouped by exact pos string.
2. 2.Keep the first occurrence
3. 3.Drop all later duplicates

This filtering is purely rule-based (no model scoring), targeting high-noise templates and low-information positives while preserving broad retrieval coverage.