patrickechohelloworld/well_formatted_benchmarks_pro

Dataset Card for wellformattedbenchmarks_pro

<!-- Provide a quick summary of the dataset. --> This is a collection of formatted benchmarks.

Dataset Details

Dataset Description

<!-- Provide a longer summary of what this dataset is. --> This repo is home to formatted versions of some famous benchmarks I created this repo because current benchmark datasets on the hub generally don't have a fixed format, which is annoying when you try to use them.

• —Language(s) (NLP): English

Dataset Sources

<!-- Provide the basic links for the dataset. -->

ARC

• —Repository: Original ARC repo
• —Demo:

<user>An astronomer observes that a planet rotates faster after a meteorite impact. Which is the most likely effect of this increase in rotation?\
<sep>A: Planetary density will decrease.</sep><sep>B: Planetary years will become longer.</sep><sep>C: Planetary days will become shorter.</sep>\
<sep>D: Planetary gravity will become stronger.</sep></user><model>C

GSM8K

• —Repository: Original GSM8K repo
• —Demo:

<user>A robe takes 2 bolts of blue fiber and half that much white fiber.  How many bolts in total does it take?</user><model>It takes 2/2=<<2/2=1>>1 bolt of white fiber
So the total amount of fabric is 2+1=<<2+1=3>>3 bolts of fabric
#### 3

HellaSwag

• —Repository: Original HellaSwag repo
• —Demo:

<user> The topic of this sentence is: Getting a haircut. Based on the topic of this sentence, finish this sentence: \
The man in the center is demonstrating a hairstyle on the person wearing the blue shirt. the man in the blue shirt\
<sep>A: is standing on the sponge cutting the hair of the person wearing the blue shirt.</sep>\
<sep>B: is doing the hairstyle with his hand and the hairspray.</sep><sep>C: sits on the chair next to the sink.</sep>\
<sep>D: is being shown eye to eye.</sep></user><model>C

MMLU

• —Repository: Original MMLU repo
• —Demo:

<user>Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q.<sep>A: 0</sep><sep>B: 4</sep><sep>C: 2</sep><sep>D: 6</sep></user><model>B

OpenBookQA

• —Repository: Original OpenBookQA repo
• —Demo:

<user> It is ture that: predators eat prey. Based on this fact, answer the following question:\
 Predators eat<sep>A: lions</sep><sep>B: humans</sep><sep>C: bunnies</sep><sep>D: grass</sep></user><model>C

TriviaQA

• —Repository: Original TriviaQA repo
• —Demo:

<user>Which American-born Sinclair won the Nobel Prize for Literature in 1930?</user>Sinclair Lewis

PIQA

• —Repository: Original PIQA repo
• —Demo

<user>The goal is: Make outdoor pillow.<sep>A: Blow into tin can and tie with rubber band.</sep>\
<sep>B: Blow into trash bag and tie with rubber band.</sep></user><model>B

WinograndeXL

• —Repository:Original Winogrande repo(XL ver. is used)
• —Demo

<user>Fill in the blank in the sentence(marked by _) with options: \
Ian volunteered to eat Dennis's menudo after already having a bowl because _ despised eating intestine.\
<sep>A: Ian</sep><sep>B: Dennis</sep></user><model>B

Uses

<!-- Address questions around how the dataset is intended to be used. -->

Direct Use

<!-- This section describes suitable use cases for the dataset. --> It's recommended to use this dataset by downloading the parquet files from main branch and load it with polars:

import polars as pl
data = pl.read_parquet('./path/to/downloaded/file').get_column('text')
...

The special tokens used in this repo include:

<user>: the beginning of prompt
</user>: the end of prompt
<model>: the beginning of response
<sep>: the beginning of an option
</sep>: the end of an option

These tokens works well with my custom tokenizer, but remember to replace them with your own special tokens like this:

# Replace with another token
text = text.replace('<model>', 'YOUR_SPECIAL_TOKEN')

# Remove the special token
text = text.replace('<model>', '')

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

Like what I mentioned above, a custom tokenizer is used to generate the files in token folder. So you need to tokenize the dataset yourself.

Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

• —Raw data from original repos are in the root directory of each subsets.
• —Formatted(but not tokenized) data files are in ./processed directory.
• —Tokenized data files are in ./token directory (and you probably don't need them, as mentioned above)

.
├── processed                <- This is the formatted data you want!
│   ├── train.parquet
│   └── validation.parquet
├── token
│   ├── train.parquet
│   └── validation.parquet
├── train.parquet         
└── validation.parquet       <- These are raw data files 

Dataset Creation

Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

The data processing is done mainly with the python scripts in the root directory (and their variants). So you can re-write these scripts based on your need to create your own formatted datasets!

Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

Please refer to the links above to see the original authors of these datasets.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. --> Please keep in mind that these datasets is for benchmarking, some of them are not suitable for SFT. Although I didn't change the content of the original datasets, it's always good practice to check them out by yourself!

Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

[More Information Needed]

Dataset Card Authors [optional]

patrickechohello_world

Dataset Card Contact

[patrickechohelloworld@gmail.com]