toolevalxm/PremiumTextDataset-TestRepo

PremiumTextDataset

<!-- markdownlint-disable first-line-h1 --> <!-- markdownlint-disable html --> <!-- markdownlint-disable no-duplicate-header -->

<div align="center"> <img src="figures/fig1.png" width="60%" alt="PremiumTextDataset" /> </div> <hr>

<div align="center" style="line-height: 1;"> <a href="LICENSE" style="margin: 2px;"> <img alt="License" src="figures/fig2.png" style="display: inline-block; vertical-align: middle;"/> </a> </div>

1. Introduction

The PremiumTextDataset has undergone significant quality improvements through rigorous data cleaning and validation pipelines. In the latest version, we have enhanced data completeness and consistency by implementing advanced deduplication algorithms and quality filters during post-processing. The dataset demonstrates outstanding performance across various quality metrics, including accuracy, completeness, and consistency. Its overall quality is now approaching that of other leading datasets.

<p align="center"> <img width="80%" src="figures/fig3.png"> </p>

Compared to the previous version, the upgraded dataset shows significant improvements in handling edge cases and reducing noise. For instance, in quality audits, the dataset's accuracy has increased from 85% in the previous version to 94.5% in the current version. This advancement stems from enhanced filtering depth during the cleaning process: in quality tests, the previous version had 12% noise rate, whereas the new version averages only 3.5% noise.

Beyond its improved data quality, this version also offers reduced duplication rate and enhanced label accuracy for supervised learning tasks.

2. Quality Assessment Results

Comprehensive Quality Metrics

<div align="center">

</div>

Overall Quality Summary

The PremiumTextDataset demonstrates strong performance across all evaluated quality categories, with particularly notable results in completeness and integrity metrics.

3. Data Access & API Platform

We offer a data portal and API for you to access PremiumTextDataset. Please check our official website for more details.

4. How to Use

Please refer to our code repository for more information about loading PremiumTextDataset locally.

Compared to previous versions, the usage recommendations for PremiumTextDataset have the following changes:

1. 1.Streaming is now supported for large-scale processing.
2. 2.It is not required to apply additional filtering for most use cases.

The data format of PremiumTextDataset-Small is identical to its base version, but it uses a compressed storage configuration. This dataset can be loaded in the same manner as the main version.

Loading Configuration

We recommend using the following configuration with a specific split.

from datasets import load_dataset

dataset = load_dataset("org/PremiumTextDataset", split="train")

Batch Size

We recommend setting the batch_size parameter to 32 for optimal performance.

Data Preprocessing

For data loading, please follow the template to create preprocessing pipelines, where {fieldname}, {transformfunc} and {output_path} are arguments.

preprocessing_template = \
"""[field name]: {field_name}
[transform begin]
{transform_func}
[transform end]
Output: {output_path}"""

5. License

This dataset is licensed under the Apache 2.0 License. The use of PremiumTextDataset is also subject to the Apache 2.0 License. The dataset supports commercial use and derivative works.

6. Contact

If you have any questions, please raise an issue on our GitHub repository or contact us at data@PremiumTextDataset.ai.