agentlans/high-quality-text

High Quality Text Dataset

A curated collection of English-language texts for AI training and research.

Sources

• —HuggingFaceFW/fineweb-edu
• —openbmb/Ultra-FineWeb
• —Zyphra/Zyda-2
• —EssentialAI/eai-taxonomy-stem-w-dclm-100b-sample
• —m-a-p/FineFineWeb

Each dataset was processed as follows:

1. 1.Split into approximately 2 000-token chunks using the LLaMA 3.1 tokenizer.
2. 2.Cleaned by normalizing spaces, punctuation, and characters, and replacing emails and phone numbers with placeholders.
3. 3.Scored using the `agentlans/GIST-all-MiniLM-L6-v2-quality-v3` classifier:
4. 4.Only chunks with a quality score greater than 1 were included.
5. 5.Removed exact duplicates.

After filtering, 100 000 chunks per source were included in the final dataset.

Clustering

Agglomerative clustering was applied using embeddings from the `Snowflake/snowflake-arctic-embed-xs` model at multiple cluster counts: 100, 200, 500, 1 000, 2 000, 5 000, 10 000, 20 000, 50 000, 100 000, and 200 000 clusters, enabling flexible dataset configurations.

Example Entry

{
  "text": "Dr. Louise Glew has been appointed the Global Lead Scientist for WWF's Global Science Team. Louise's research focuses on understanding the social and ecological impacts of conservation interventions [...]",
  "quality": 2.0699,
  "source": "openbmb/Ultra-FineWeb"
}

Limitations

• —Primarily focuses on academic, educational, and pedagogical content intended for a general audience.
• —May include outdated, unreliable, or controversial information (such as self-published material, pseudoscience, or conspiracy theories).
• —Quality scores evaluate syntax and tone, but do not guarantee factual accuracy.
• —Occasional repetition may occur (for example, dictionary entries or geographic distance calculations).
• —Entries might be interrupted mid-word or mid-sentence.

Licence

Provided under the Open Data Commons Attribution License (ODC-BY).