hotchpotch/cornstack-samples

1---2dataset_info:3- config_name: go-v1-hard-negatives-100k4  features:5  - name: query6    dtype: string7  - name: pos8    dtype: string9  - name: negs10    list: string11  splits:12  - name: train13    num_examples: 8753614- config_name: go-v1-pair-2M15  features:16  - name: query17    dtype: string18  - name: pos19    dtype: string20  splits:21  - name: train22    num_examples: 153231923- config_name: java-v1-hard-negatives-100k24  features:25  - name: query26    dtype: string27  - name: pos28    dtype: string29  - name: negs30    list: string31  splits:32  - name: train33    num_examples: 8090634- config_name: java-v1-pair-2M35  features:36  - name: query37    dtype: string38  - name: pos39    dtype: string40  splits:41  - name: train42    num_examples: 145665143- config_name: javascript-v1-hard-negatives-100k44  features:45  - name: query46    dtype: string47  - name: pos48    dtype: string49  - name: negs50    list: string51  splits:52  - name: train53    num_examples: 7948454- config_name: javascript-v1-pair-2M55  features:56  - name: query57    dtype: string58  - name: pos59    dtype: string60  splits:61  - name: train62    num_examples: 129874563- config_name: php-v1-hard-negatives-100k64  features:65  - name: query66    dtype: string67  - name: pos68    dtype: string69  - name: negs70    list: string71  splits:72  - name: train73    num_examples: 7494574- config_name: php-v1-pair-2M75  features:76  - name: query77    dtype: string78  - name: pos79    dtype: string80  splits:81  - name: train82    num_examples: 131130283- config_name: python-v1-hard-negatives-100k84  features:85  - name: query86    dtype: string87  - name: pos88    dtype: string89  - name: negs90    list: string91  splits:92  - name: train93    num_examples: 9667094- config_name: python-v1-pair-2M95  features:96  - name: query97    dtype: string98  - name: pos99    dtype: string100  splits:101  - name: train102    num_examples: 1779891103- config_name: ruby-v1-hard-negatives-100k104  features:105  - name: query106    dtype: string107  - name: pos108    dtype: string109  - name: negs110    list: string111  splits:112  - name: train113    num_examples: 67937114- config_name: ruby-v1-pair-2M115  features:116  - name: query117    dtype: string118  - name: pos119    dtype: string120  splits:121  - name: train122    num_examples: 1145414123configs:124- config_name: go-v1-hard-negatives-100k125  data_files:126  - split: train127    path: go-v1-hard-negatives-100k/train-*128- config_name: go-v1-pair-2M129  data_files:130  - split: train131    path: go-v1-pair-2M/train-*132- config_name: java-v1-hard-negatives-100k133  data_files:134  - split: train135    path: java-v1-hard-negatives-100k/train-*136- config_name: java-v1-pair-2M137  data_files:138  - split: train139    path: java-v1-pair-2M/train-*140- config_name: javascript-v1-hard-negatives-100k141  data_files:142  - split: train143    path: javascript-v1-hard-negatives-100k/train-*144- config_name: javascript-v1-pair-2M145  data_files:146  - split: train147    path: javascript-v1-pair-2M/train-*148- config_name: php-v1-hard-negatives-100k149  data_files:150  - split: train151    path: php-v1-hard-negatives-100k/train-*152- config_name: php-v1-pair-2M153  data_files:154  - split: train155    path: php-v1-pair-2M/train-*156- config_name: python-v1-hard-negatives-100k157  data_files:158  - split: train159    path: python-v1-hard-negatives-100k/train-*160- config_name: python-v1-pair-2M161  data_files:162  - split: train163    path: python-v1-pair-2M/train-*164- config_name: ruby-v1-hard-negatives-100k165  data_files:166  - split: train167    path: ruby-v1-hard-negatives-100k/train-*168- config_name: ruby-v1-pair-2M169  data_files:170  - split: train171    path: ruby-v1-pair-2M/train-*172license: apache-2.0173---174# cornstack-samples175 176🚧 This dataset is under active development and may change.177 178Filtered CoRNStack sample subsets for code retrieval training.179 180Source dataset and paper:181- CoRNStack collection: https://huggingface.co/collections/nomic-ai/cornstack182- CoRNStack paper: https://huggingface.co/papers/2412.01007183 184Note: the original CoRNStack collection is a much larger dataset family for code search training.185If you need large-scale data (not samples), please refer to the original CoRNStack collection above.186 187## What This Release Contains188 189This release keeps the original subset layout (6 languages x pair + hard-negatives) and applies deterministic rule-based filtering.190 191In this revision, query-level deduplication and positive-document (`pos`) deduplication are applied per subset.192For both rules, if duplicates exist, only the first row is kept.193 194## Config Layout And Schema195 196Each language is published as two configs with split `train`:197- `{lang}-v1-pair-2M`198- `{lang}-v1-hard-negatives-100k`199 200Schema:201- Pair configs: `query`, `pos`202- Hard-negative configs: `query`, `pos`, `negs` (list[string])203 204## Subsets And Row Counts (After Filter + Query Dedup + Pos Dedup)205 206Note: subset names keep the original `2M` / `100k` naming convention for compatibility, but207actual row counts vary by language and can be substantially lower after quality filtering and208deduplication.209 210| Subset (config name) | split | num_examples |211| --- | --- | ---: |212| `go-v1-pair-2M` | `train` | 1,532,319 |213| `go-v1-hard-negatives-100k` | `train` | 87,536 |214| `java-v1-pair-2M` | `train` | 1,456,651 |215| `java-v1-hard-negatives-100k` | `train` | 80,906 |216| `javascript-v1-pair-2M` | `train` | 1,298,745 |217| `javascript-v1-hard-negatives-100k` | `train` | 79,484 |218| `php-v1-pair-2M` | `train` | 1,311,302 |219| `php-v1-hard-negatives-100k` | `train` | 74,945 |220| `python-v1-pair-2M` | `train` | 1,779,891 |221| `python-v1-hard-negatives-100k` | `train` | 96,670 |222| `ruby-v1-pair-2M` | `train` | 1,145,414 |223| `ruby-v1-hard-negatives-100k` | `train` | 67,937 |224 225Total rows:226- Pair: 8,524,322227- Hard-negatives: 487,478228- Overall: 9,011,800229 230## Dedup Impact (Pos Dedup Stage)231 232The table below shows the `pos`-dedup impact on top of the previous filtered + query-deduped data.233 234| Subset | before | after | removed | removed_ratio |235| --- | ---: | ---: | ---: | ---: |236| `go-v1-pair-2M` | 1,541,111 | 1,532,319 | 8,792 | 0.57% |237| `go-v1-hard-negatives-100k` | 87,647 | 87,536 | 111 | 0.13% |238| `java-v1-pair-2M` | 1,491,655 | 1,456,651 | 35,004 | 2.35% |239| `java-v1-hard-negatives-100k` | 81,657 | 80,906 | 751 | 0.92% |240| `javascript-v1-pair-2M` | 1,310,965 | 1,298,745 | 12,220 | 0.93% |241| `javascript-v1-hard-negatives-100k` | 79,684 | 79,484 | 200 | 0.25% |242| `php-v1-pair-2M` | 1,343,442 | 1,311,302 | 32,140 | 2.39% |243| `php-v1-hard-negatives-100k` | 75,632 | 74,945 | 687 | 0.91% |244| `python-v1-pair-2M` | 1,807,480 | 1,779,891 | 27,589 | 1.53% |245| `python-v1-hard-negatives-100k` | 97,147 | 96,670 | 477 | 0.49% |246| `ruby-v1-pair-2M` | 1,175,219 | 1,145,414 | 29,805 | 2.54% |247| `ruby-v1-hard-negatives-100k` | 68,382 | 67,937 | 445 | 0.65% |248 249Stage totals:250- Pair removed by pos dedup: 145,550 (1.68%)251- Hard-negatives removed by pos dedup: 2,671 (0.54%)252- Overall removed by pos dedup: 148,221 (1.62%)253 254## Quick Usage255 256```python257from datasets import load_dataset258 259pair_ds = load_dataset("hotchpotch/cornstack-samples", "python-v1-pair-2M", split="train")260hard_ds = load_dataset("hotchpotch/cornstack-samples", "python-v1-hard-negatives-100k", split="train")261 262print(pair_ds.column_names, len(pair_ds))263print(hard_ds.column_names, len(hard_ds))264```265 266## License267 268This dataset follows CoRNStack and is released under **Apache-2.0**.269 270## Citation And Attribution271 272If you use this dataset, please cite and attribute CoRNStack:273- Paper: https://huggingface.co/papers/2412.01007274- Collection: https://huggingface.co/collections/nomic-ai/cornstack275 276## Noise Filtering Algorithm (Rule-based)277 278The following deterministic rules are applied before publishing this release.279 2801. Prefix-based noisy query removal281A row is dropped if `query` starts with any of the following prefixes:282- `TODO`283- `GET /`284- `POST /`285- `PUT /`286- `DELETE /`287- `Display a listing of the resource.`288- `Store a newly created resource in storage.`289- `Show the form for editing the specified resource.`290- `Update the specified resource in storage.`291- `Show the form for creating a new resource.`292- `Remove the specified resource from storage.`293- `Display the specified resource.`294- `Transform the resource into an array.`295- `Autogenerated method stub`296- `Auto generated`297- `this down() migration is autogenerated`298- `this up() migration is autogenerated`299- `"/ renamed from:"`300- `"/ access modifiers changed from:"`301 3022. Minimum positive-document length303A row is dropped if the positive side text is shorter than 30 characters.304- Pair configs: `pos` length >= 30 required305- Hard-negative configs: `pos` length >= 30 required306 3073. Hard-negative size constraint308For hard-negative configs in this release, each row keeps exactly seven negatives after normalization (`len(negs) = 7`, equivalent to `min_negs = 7`).309 3104. Query-level deduplication311Within each subset split, rows are grouped by exact `query` string.312- Keep the first occurrence313- Drop all later duplicates314 3155. Positive-document (`pos`) deduplication316Within each subset split, rows are grouped by exact `pos` string.317- Keep the first occurrence318- Drop all later duplicates319 320This filtering is purely rule-based (no model scoring), targeting high-noise templates and low-information positives while preserving broad retrieval coverage.321