agentlans/low-readability-text

Low Readability Text Dataset

This dataset consists of high-complexity English web text with an estimated readability at or above the U.S. Grade 12 level. The content typically features advanced, highly technical prose or verbose syntactical structures, making it well-suited for researching complex language understanding and automation.

Primary Use Cases

• —Text Simplification: Training and evaluating models to translate complex text into plain English.
• —Information Extraction (IE): Benchmarking NLP systems on dense, complex text structures that general human readers find difficult to parse.
• —Readability Assessment: Training classifiers to recognize specialized, collegiate, or advanced reading levels.

Dataset Creation & Methodology

1. Extraction & Chunking

• —Text was ingested from major web corpuses and broken into chunks of 1024 GPT-4 tokens, featuring a 10-token overlap between consecutive segments to maintain contextual continuity.
• —50,000 chunks per source dataset were initially isolated where the GIST-small-readability classifier scored the text at ≥ 12 (equivalent to U.S. high school senior/university freshman level or higher).

2. Quality Filtering

• —The initial pool was filtered using the agentlans/GIST-all-MiniLM-L6-v2-quality-v3 model. Only chunks with a quality score > 1.0 were retained.
• —In addition, Markov chain-generated text were filtered out using agentlans/GIST-small-markov-slop-detector.

3. Clustering

• —Post-filtering, data reduction and organization were performed via Agglomerative Clustering utilizing MongoDB/mdbr-leaf-mt embeddings to group similar semantic contexts.

Data Splits & Source Composition

The current primary split is sample_k100000. The distribution of source rows is outlined below:

• —Low retention due to high prevalence of spam

Dataset Structure

Features

• —text (String): The 1024-token text block.
• —grade (Float): The assigned reading grade level (minimum 12.0).
• —source (String): The originating dataset identifier.

Example Row

{
  "text": "the patents , Galium-Arsenide 935 nm 730 nm?270-275nm? 310-320nm and 760nm (H2O) 230-240nm (N2) 155-160nm (o2) Water vapor-nitrogen absorption at CO(2) laser frequencies. Peterson JC, Thomas ME, Nordstrom RJ, Damon EK, Long RK. \"...a series of pressure-broadened water vapor absorption measurements at 27 CO(2) laser frequencies between 935 cm(-1) and 1082 cm(-1)\" D20 (Deterium heavy water) - shift H2O from 760nm to 1000nm Balmer series or Balmer lines in atomic physics, is the designation of one of a set of six different named series describing the spectral line emissions of the hydrogen atom...",
  "grade": 12.0,
  "source": "Zyphra/Zyda-2"
}

Limitations & Considerations

⚠️ Usage Warnings:

• —Boundary Fragmentation: Because the dataset relies on rigid token-based windowing (1024 tokens), text blocks may start or end mid-sentence or mid-paragraph.
• —Domain Generality: The texts represent a broad, general-purpose web crawl. While complex, the text does not strictly represent deep, peer-reviewed domains like academic law or professional medicine.
• —Web Noise & Accuracy: The corpus stems from unverified internet pages. It contains messy syntax, optical character recognition/scraping artifacts, and may include scientifically or historically inaccurate claims.

Licence

This dataset is made available under the Open Data Commons Attribution License (ODC-By v1.0).