agentlans/high-quality-english-sentences

High-Quality English Sentences

Dataset Description

This dataset contains a collection of high-quality English sentences sourced from C4 and FineWeb (not FineWeb-Edu). The sentences have been carefully filtered and processed to ensure quality and uniqueness.

"High-quality" means they're legible English and not spam, although they may still have spelling and grammar errors.

Source Data

Before filtering:

• —C4: 1 million sentences
• —FineWeb: 1 million sentences

Data Processing

1. 1.Initial Sentence Filtering:
2. 2.Quality score > 0.5 with agentlans/deberta-v3-xsmall-quality classifier
3. 3.Length >= 20 characters

1. 1.Additional Filtering:
2. 2.Removed sentences not starting with a capital letter
3. 3.Removed sentences with unmatched parentheses or brackets

1. 1.Deduplication:
2. 2.Combined sentences from both sources
3. 3.Filtered for unique sentences (exact match)

1. 1.Train-Test Split:
2. 2.Train set: 90% of data
3. 3.Test set: 10% of data

Data Fields

No data field. Just one sentence per line as a string.

Data Splits

Dataset Creation

• —Curation Rationale: To create a high-quality dataset of diverse English sentences for various NLP tasks.
• —Source Contributions: Equal initial contributions from C4 and FineWeb, with subsequent filtering and processing.

Considerations for Using the Data

• —The dataset deliberately uses FineWeb and C4 instead of FineWeb-Edu in order to broaden beyond educational and academic content. However, it's still biased towards those genres.
• —Titles and citations are considered "sentences".
• —The filtering process may have introduced some biases, such as favouring more formal writing styles and omitting lists due to the capitalization requirement.