CoolFace
Datasetpublic

toolevalxm/MySuperDataset-TestRepo

MySuperDataset 1. Introduction MySuperDataset is a comprehensive multi-domain text corpus designed for training and evaluating language models. This dataset has been carefully curated through multiple iterations to ensure high quality, diversity, and representativeness across various domains including science, technology, arts, and everyday conversations. The dataset includes over 10 million samples spanning 15 different quality… See the full description on the dataset page: https://huggingface.co/datasets/toolevalxm/MySuperDataset-TestRepo.

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face
0likes4downloads
Dataset Card

MySuperDataset

<!-- markdownlint-disable first-line-h1 --> <!-- markdownlint-disable html --> <!-- markdownlint-disable no-duplicate-header -->

<div align="center"> <img src="figures/dataset_overview.png" width="60%" alt="MySuperDataset" /> </div> <hr>

<div align="center" style="line-height: 1;"> <a href="LICENSE" style="margin: 2px;"> <img alt="License" src="figures/license_badge.png" style="display: inline-block; vertical-align: middle;"/> </a> </div>

1. Introduction

MySuperDataset is a comprehensive multi-domain text corpus designed for training and evaluating language models. This dataset has been carefully curated through multiple iterations to ensure high quality, diversity, and representativeness across various domains including science, technology, arts, and everyday conversations.

<p align="center"> <img width="80%" src="figures/quality_chart.png"> </p>

The dataset includes over 10 million samples spanning 15 different quality metrics. Through our iterative curation process, we have significantly improved data quality while maintaining diversity. The latest version shows a 25% improvement in overall quality scores compared to the initial release.

Our curation process focuses on removing harmful content, deduplication, and ensuring balanced representation across topics and demographics.

2. Quality Metrics

Comprehensive Quality Assessment

<div align="center">

MetricBaselineVersion1Version2MySuperDataset
Content QualityFactual Accuracy0.7200.7450.7680.829
Grammar Score0.8500.8720.8900.879
Coherence0.7800.7950.8120.873
Diversity MetricsTopic Coverage0.6500.6800.7050.773
Vocabulary Richness0.7200.7380.7550.806
Style Variety0.6800.7020.7200.768
Source Diversity0.5900.6250.6580.743
Safety & EthicsToxicity Filter0.9200.9350.9480.963
Bias Mitigation0.7800.8050.8250.847
Privacy Protection0.8500.8720.8880.907
Copyright Compliance0.9100.9250.9380.960
Technical QualityDeduplication Rate0.8800.9050.9220.935
Format Consistency0.8200.8450.8650.880
Encoding Quality0.9500.9620.9700.970
Metadata Completeness0.7500.7780.8000.828

</div>

Overall Quality Summary

MySuperDataset demonstrates exceptional quality across all evaluated metrics, with particularly strong performance in safety filtering and content quality dimensions.

3. Dataset Access & API

We provide direct dataset access through our platform and API endpoints. Please visit our official website for documentation and access tokens.

4. How to Use

Please refer to our documentation for detailed usage instructions.

Loading the Dataset

python
from datasets import load_dataset

dataset = load_dataset("MySuperDataset")

Data Format

Each sample contains:

  • —text: The main text content
  • —domain: The topic domain classification
  • —quality_score: Pre-computed quality score
  • —source: Original data source identifier

Recommended Usage

We recommend filtering samples with quality_score >= 0.8 for training language models.

5. License

This dataset is released under the Apache 2.0 License. Commercial use is permitted with proper attribution.

6. Contact

For questions or issues, please open a GitHub issue or contact us at data@mysuperdataset.ai.