AryanAnuj/processed_dataset_orca-math-word-problems-200k
Dataset Description: This dataset contains data that has undergone two preprocessing steps: Removal of Instructions with Less Than 100 Tokens in Response: Instructions with less than 100 tokens in the response have been removed from the dataset. This preprocessing step helps to ensure that the dataset contains substantial and informative responses. Data Deduplication by Grouping Using Cosine Similarity (Threshold > 0.95): Data deduplication has been performed by grouping similar instances… See the full description on the dataset page: https://huggingface.co/datasets/AryanAnuj/processed_dataset_orca-math-word-problems-200k.
Dataset Description:
This dataset contains data that has undergone two preprocessing steps:
Removal of Instructions with Less Than 100 Tokens in Response: Instructions with less than 100 tokens in the response have been removed from the dataset. This preprocessing step helps to ensure that the dataset contains substantial and informative responses.
Data Deduplication by Grouping Using Cosine Similarity (Threshold > 0.95): Data deduplication has been performed by grouping similar instances together using cosine similarity. Instances with a cosine similarity greater than 0.95 have been considered duplicates and grouped accordingly. This preprocessing step helps to remove redundant or highly similar instances from the dataset, improving its quality and reducing redundancy.
Now it has distinct question and thier respective answers .
It's Ready to train a Large Language Model on Math word problem.