Datasetpublic

toolevalxm/SuperDataset-QualityRelease

SuperDataset 1. Introduction SuperDataset represents a new generation of curated training data for natural language processing tasks. Our latest release incorporates advanced data curation techniques including automated quality filtering, cross-validation with multiple annotators, and comprehensive bias detection. The dataset demonstrates exceptional quality metrics across all evaluation dimensions. Compared to the previous version… See the full description on the dataset page: https://huggingface.co/datasets/toolevalxm/SuperDataset-QualityRelease.

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes33downloads

Dataset Card

SuperDataset

1. Introduction

SuperDataset represents a new generation of curated training data for natural language processing tasks. Our latest release incorporates advanced data curation techniques including automated quality filtering, cross-validation with multiple annotators, and comprehensive bias detection. The dataset demonstrates exceptional quality metrics across all evaluation dimensions.

Compared to the previous version, the upgraded dataset shows significant improvements in data quality. For instance, annotation accuracy has increased from 82% in the previous version to 94.5% in the current version. This advancement stems from our enhanced multi-stage annotation pipeline and improved quality control mechanisms.

Beyond its improved quality metrics, this version also offers reduced noise levels and enhanced domain coverage.

2. Quality Metrics

Comprehensive Quality Results

	Quality Metric	Dataset1	Dataset2	Dataset1-v2	SuperDataset
Data Integrity	Data Completeness	0.820	0.845	0.860	0.858
	Annotation Accuracy	0.875	0.890	0.901	0.791
	Label Consistency	0.790	0.810	0.825	0.820
Format Quality	Format Validity	0.920	0.935	0.945	0.870
	Schema Compliance	0.850	0.870	0.880	0.861
	Token Distribution	0.780	0.795	0.810	0.780
	Sample Validity	0.815	0.830	0.845	0.836
Content Quality	Diversity Index	0.720	0.745	0.760	0.750
	Balance Ratio	0.680	0.705	0.720	0.720
	Coverage Score	0.750	0.770	0.785	0.783
	Language Quality	0.830	0.855	0.870	0.852
Safety Metrics	Noise Level	0.910	0.925	0.935	0.820
	Duplicate Rate	0.880	0.900	0.915	0.863
	Privacy Check	0.950	0.960	0.970	0.891
	Bias Detection	0.840	0.860	0.875	0.846

</div>

Overall Quality Summary

SuperDataset demonstrates superior quality across all evaluated metric categories, with particularly notable results in data integrity and safety metrics.

3. Data Access & API Platform

We offer a data browser interface and API for you to access SuperDataset. Please check our official website for more details.

4. How to Use Locally

Please refer to our code repository for more information about using SuperDataset locally.

Compared to previous versions, the usage recommendations for SuperDataset have the following changes:

1.Streaming download is supported.
2.It is not required to manually validate data format - automatic validation is included.

Loading the Dataset

We recommend using the following code pattern:

python

from datasets import load_dataset

dataset = load_dataset("SuperDataset")

For example,

python

from datasets import load_dataset

dataset = load_dataset("SuperDataset", split="train")
print(dataset[0])

Batch Size

We recommend setting the batch_size parameter to 32 for optimal loading performance.

Data Filtering

For data filtering, please follow the template below, where {filtercolumn}, {filtervalue} and {operation} are arguments.

filter_template = \
"""dataset.filter(lambda x: x['{filter_column}'] {operation} {filter_value})"""

5. License

This dataset is licensed under the Apache 2.0 License. The use of SuperDataset is also subject to the Apache 2.0 License. The dataset supports commercial use and derivative works.

6. Contact

If you have any questions, please raise an issue on our GitHub repository or contact us at contact@SuperDataset.ai.