EssentialAI/essential-web-v1.0

1---2license: odc-by3size_categories:4- 10B<n<100B5---6# 🌐 Essential-Web: Complete 24-Trillion Token Dataset7 8[🏆 Website](https://www.essential.ai/)  |  [🖥️ Code](https://github.com/Essential-AI/eai-taxonomy)  |  [📖 Paper](https://huggingface.co/papers/2506.14111) | [☁️ AWS](https://registry.opendata.aws/eai-essential-web-v1/)9 10## 📋 Dataset Description11 12Essential-Web is a 24-trillion-token web dataset with document-level metadata designed for flexible dataset curation. The dataset provides metadata including subject matter classification, web page type, content complexity, and document quality scores for each of the 23.6 billion documents.13 14Researchers can filter and curate specialized datasets using the provided metadata, reducing the need for custom preprocessing pipelines and domain-specific classifiers.15 16## 🔍 Free Decimal Correspondence (FDC) Taxonomy17 18Essential-Web uses the Free Decimal Correspondence, a Dewey Decimal-inspired open taxonomy with 12 main categories for classifying web content. This systematic approach enables precise domain filtering and dataset curation.19 20For help navigating FDC codes, see: https://www.librarything.com/mds21 22## ⚙️ Dataset Creation23 24Essential-Web was created using a comprehensive processing pipeline starting from Common Crawl data:25 26### 📥 Source Data27- **DCLM Pool**: 89 resiliparse-extracted Common Crawl WARC snapshots (CC-MAIN-2013-20 to CC-MAIN-2022-49)28- **Additional Snapshots**: 12 additional snapshots extracted from CC-MAIN-2023-06 to CC-MAIN-2024-38 using resiliparse29- **Total**: 101 Common Crawl snapshots processed30 31### 🔧 Processing Pipeline321. **Document ID Generation**: Using xxhash.xxh3_64_intdigest for unique document identification332. **Global Deduplication**: Hash-based deduplication across all 101 snapshots343. **Minhash LSH Deduplication**: Snapshot-level deduplication with Jaccard threshold of 0.7 (14 bands, 9 rows per band)354. **Quality Annotation**: Statistical and model-based quality signals using RedPajama-Data-V2 pipeline variant, including DCLM-baseline fastText classifier365. **Quality Filtering**: Manual tuned filters to retain high-quality English documents while preserving math and code content376. **Taxonomy Labeling**: Classification of every document using EAI-Taxonomy-0.5b (~90,000 AMD MI300x GPU-hours)38 39## 🎯 Performance & Validation40 41We've curated example domain-specific datasets from Essential-Web using simple metadata filters, showing competitive performance relative to top performing web-curated datasets:42 43- 🧮 **Math**: within 8.0% of web-curated baselines44- 💻 **Web Code**: 14.3% above web-curated baselines  45- 🔬 **STEM**: 24.5% above web-curated baselines46- 🩺 **Medical**: 8.6% above web-curated baselines47 48*Note: These represent initial examples with significant room for further curation and improvement. Comparisons are against web-sourced datasets rather than specialized synthetic datasets.*49 50## 🚀 Related Datasets & Models51 52### Domain-Specific Datasets53We've curated high-quality domain-specific datasets from Essential-Web:54 55- **Math**: [EssentialAI/eai-taxonomy-math-w-fm](https://huggingface.co/datasets/EssentialAI/eai-taxonomy-math-w-fm)56- **Code**: [EssentialAI/eai-taxonomy-code-w-dclm](https://huggingface.co/datasets/EssentialAI/eai-taxonomy-code-w-dclm)57- **Medical**: [EssentialAI/eai-taxonomy-med-w-dclm](https://huggingface.co/datasets/EssentialAI/eai-taxonomy-med-w-dclm)58- **STEM**: [EssentialAI/eai-taxonomy-stem-w-dclm](https://huggingface.co/datasets/EssentialAI/eai-taxonomy-stem-w-dclm)59 60### Classification Model61- **EAI-Taxonomy-0.5b**: [EssentialAI/eai-taxonomy-0.5b](https://huggingface.co/EssentialAI/eai-taxonomy-0.5b) - The efficient classifier used to label Essential-Web documents62 63## 🎯 Intended Use64 65Essential-Web enables researchers to:66- 🚀 **Rapid Curation**: Create multi-billion-token domain-specific datasets in minutes using SQL-like filters67- 🔍 **Flexible Exploration**: Explore web content across subjects, quality levels, and content types68- 🏗️ **Custom Pipelines**: Build specialized training corpora without custom classification infrastructure69- 🔄 **Iterative Improvement**: Easily modify and refine dataset composition based on training results70- 📊 **Quality Control**: Filter out low-quality content (ads, product listings) while preserving reasoning-dense documents71 72# Dataset Schema Documentation73 74## Overview75 76This dataset contains web-crawled text data with comprehensive metadata, quality signals, and taxonomic classifications. Each record represents a document extracted from web archives with detailed provenance tracking and quality assessment metrics.77 78## Core Fields79 80| Field | Type | Description | Path |81|-------|------|-------------|------|82| `id` | `Int64` | Unique identifier based on document hash | `id` |83| `text` | `String` | The main textual content of the document | `text` |84 85## EAI Taxonomy Classification86 87Comprehensive hierarchical classification system with primary and secondary labels - the most important feature of this dataset. The taxonomy is designed to provide detailed subject categorization, document type identification, content quality assessment, and extraction quality indicators.88 89<details>90<summary><strong>Free Decimal Correspondence (FDC)</strong></summary>91 92A Dewey Decimal-inspired classification system with 3-level hierarchical labels. The FDC provides nested categories where each successive level refines its parent category. It's designed to be compatible with the Dewey Decimal System for library cataloging.93 94**Level Structure:**95- **Level 1**: Top-level categories (0-9) covering broad subject areas like General works, Philosophy, Religion, Social Sciences, etc.96- **Level 2**: Sub-divisions (00-99) that refine Level 1 categories97- **Level 3**: Specific categories (000-999) that further refine Level 2 categories98 99| Component | Description | Path |100|-----------|-------------|------|101| Primary Code | Main classification code | `eai_taxonomy.free_decimal_correspondence.primary.code` |102| Primary Level 1 | Top-level category (0=General works, 1=Philosophy, 2=Religion, 3=Social Sciences, 4=Language, 5=Science, 6=Technology, 7=Arts, 8=Literature, 9=History/Geography) | `eai_taxonomy.free_decimal_correspondence.primary.labels.level_1` |103| Primary Level 2 | Mid-level category | `eai_taxonomy.free_decimal_correspondence.primary.labels.level_2` |104| Primary Level 3 | Specific category | `eai_taxonomy.free_decimal_correspondence.primary.labels.level_3` |105| Secondary Code | Alternative classification code | `eai_taxonomy.free_decimal_correspondence.secondary.code` |106| Secondary Level 1 | Alternative top-level category | `eai_taxonomy.free_decimal_correspondence.secondary.labels.level_1` |107| Secondary Level 2 | Alternative mid-level category | `eai_taxonomy.free_decimal_correspondence.secondary.labels.level_2` |108| Secondary Level 3 | Alternative specific category | `eai_taxonomy.free_decimal_correspondence.secondary.labels.level_3` |109 110We recommend this viewer for easily navigating the FDC categories when curating filters: https://www.librarything.com/mds111 112</details>113 114<details>115<summary><strong>Bloom's Taxonomy Integration</strong></summary>116 117Based on Anderson and Krathwohl's 2001 revision of Bloom's Taxonomy of Educational Objectives, providing two complementary categorization dimensions for educational content analysis.118 119### Knowledge Domain120Categorizes the type of knowledge demonstrated in the document:121 122| Component | Description | Path |123|-----------|-------------|------|124| Primary Code | Main knowledge domain code | `eai_taxonomy.bloom_knowledge_domain.primary.code` |125| Primary Label | Main knowledge domain label | `eai_taxonomy.bloom_knowledge_domain.primary.label` |126| Secondary Code | Alternative knowledge domain code | `eai_taxonomy.bloom_knowledge_domain.secondary.code` |127| Secondary Label | Alternative knowledge domain label | `eai_taxonomy.bloom_knowledge_domain.secondary.label` |128 129**Possible Values:**130| Code | Label | Description |131|------|-------|-------------|132| `-1` | Abstain | Unable to determine |133| `1` | Factual | Basic elements to learn or solve problems |134| `2` | Conceptual | Interrelationships between basic elements within larger context |135| `3` | Procedural | Methods and techniques in the discipline |136| `4` | Metacognitive | Awareness of how learning works in relation to oneself |137 138### Cognitive Processing Level139Assesses the learning and thinking skill levels demonstrated by the document author:140 141| Component | Description | Path |142|-----------|-------------|------|143| Primary Code | Main cognitive process code | `eai_taxonomy.bloom_cognitive_process.primary.code` |144| Primary Label | Main cognitive process label | `eai_taxonomy.bloom_cognitive_process.primary.label` |145| Secondary Code | Alternative cognitive process code | `eai_taxonomy.bloom_cognitive_process.secondary.code` |146| Secondary Label | Alternative cognitive process label | `eai_taxonomy.bloom_cognitive_process.secondary.label` |147 148**Possible Values:**149| Code | Label | Description |150|------|-------|-------------|151| `-1` | Abstain | Unable to determine |152| `1` | Remember | Retrieve relevant knowledge from memory |153| `2` | Understand | Determine meaning of instructional messages |154| `3` | Apply | Use a procedure in a given situation |155| `4` | Analyze | Break materials into components and determine relationships |156| `5` | Evaluate | Make judgments based on criteria and standards |157| `6` | Create | Create new or original work |158 159</details>160 161<details>162<summary><strong>Document Characteristics</strong></summary>163 164### Document Type v1165In-house classification of common web document types and formats:166 167| Component | Description | Path |168|-----------|-------------|------|169| Primary Code | Main document type code | `eai_taxonomy.document_type_v1.primary.code` |170| Primary Label | Main document type label | `eai_taxonomy.document_type_v1.primary.label` |171| Secondary Code | Alternative document type code | `eai_taxonomy.document_type_v1.secondary.code` |172| Secondary Label | Alternative document type label | `eai_taxonomy.document_type_v1.secondary.label` |173 174**Possible Values:**175| Code | Label | Examples |176|------|-------|----------|177| `-1` | Abstain | Unable to classify |178| `1` | News/Editorial | CNN articles, opinion columns |179| `2` | Academic/Research | ArXiv papers, research articles |180| `3` | Reference/Encyclopedic/Educational | FAQs, Wikipedia entries |181| `4` | Code/Software | GitHub repos, code examples |182| `5` | Social/Forum | Conversation threads, Q&A boards |183| `6` | Promotional/Advertisement | Product pages, calls to action |184| `7` | Search/Directory/Bibliography | Link pages, search results |185| `8` | Adult/Pornographic | Adult content |186| `9` | Personal/Misc | Blogs, user profiles |187| `10` | Machine-Generated | Lorem ipsum, garbled text |188| `11` | Legal/Regulatory | Contracts, terms of service |189| `12` | Government/Political | Legislation, press releases |190| `13` | Literary/Creative | Poems, short stories |191| `14` | Reviews/Critiques | Film critiques, product reviews |192| `15` | E-Commerce/Marketplace | eBay listings, Amazon pages |193| `16` | Images/Videos/Audio | YouTube videos, Imgur pages |194| `17` | Other/Unclassified | Documents that resist classification |195 196### Document Type v2197Updated classification based on WebOrganizer taxonomy with refined categories for improved document classification accuracy:198 199| Component | Description | Path |200|-----------|-------------|------|201| Primary Code | Main document type code (v2) | `eai_taxonomy.document_type_v2.primary.code` |202| Primary Label | Main document type label (v2) | `eai_taxonomy.document_type_v2.primary.label` |203| Secondary Code | Alternative document type code (v2) | `eai_taxonomy.document_type_v2.secondary.code` |204| Secondary Label | Alternative document type label (v2) | `eai_taxonomy.document_type_v2.secondary.label` |205 206**Complete Value Mapping:**207| Code | Label | Examples |208|------|-------|----------|209| `-1` | Abstain | Documents requiring human review |210| `1` | About (Org.) | Company about pages, mission statements |211| `2` | About (Personal) | Personal bios, LinkedIn profiles |212| `3` | Academic Writing | Research papers, abstracts, dissertations |213| `4` | Audio Transcript | Interview transcripts, court records, captions |214| `5` | Comment Section | Reddit threads, blog comments |215| `6` | Content Listing | Site maps, product catalogs, directory listings |216| `7` | Creative Writing | Song lyrics, novel excerpts, poetry |217| `8` | Documentation | API docs, README files, user manuals |218| `9` | FAQ | FAQ pages, Q&A lists |219| `10` | Knowledge Article | Wikipedia articles, Britannica entries |220| `11` | Legal Notices | Privacy policies, license agreements, terms of service |221| `12` | Listicle | Buzzfeed-style articles, "Top 10" lists |222| `13` | News (Org.) | Government blog posts, corporate announcements |223| `14` | News Article | Newspaper articles, CNN content, breaking news |224| `15` | Nonfiction Writing | Editorials, obituaries, memoirs, opinion pieces |225| `16` | Personal Blog | Personal journals, diary entries, lifestyle blogs |226| `17` | Product Page | Product descriptions, course offerings, sales pages |227| `18` | Q&A Forum | Quora posts, Stack Exchange discussions |228| `19` | Spam / Ads | SEO keyword stuffing, promotional spam |229| `20` | Structured Data | Datasheets, glossaries, JSON files, databases |230| `21` | Customer Support | Help articles, troubleshooting guides |231| `22` | Truncated | Paywalled sites, image galleries, partial content |232| `23` | Tutorial | Cooking recipes, WikiHow pages, step-by-step guides |233| `24` | User Review | Yelp reviews, TripAdvisor feedback, product reviews |234| `25` | Other/Unclassified | Miscellaneous documents not fitting other categories |235 236### Extraction Artifacts237Assessment of technical extraction quality, identifying issues from HTML-to-text conversion:238 239| Component | Description | Path |240|-----------|-------------|------|241| Primary Code | Main extraction artifact code | `eai_taxonomy.extraction_artifacts.primary.code` |242| Primary Label | Main extraction artifact label | `eai_taxonomy.extraction_artifacts.primary.label` |243| Secondary Code | Alternative extraction artifact code | `eai_taxonomy.extraction_artifacts.secondary.code` |244| Secondary Label | Alternative extraction artifact label | `eai_taxonomy.extraction_artifacts.secondary.label` |245 246**Possible Values:**247| Code | Label | Description |248|------|-------|-------------|249| `-1` | Abstain | Unable to determine |250| `0` | No Artifacts | Clean text with no leftover HTML or irrelevant elements |251| `1` | Leftover HTML | HTML/code artifacts remaining after extraction |252| `2` | Text Extraction Errors | Broken math expressions, encoding errors, improperly parsed tables |253| `3` | Irrelevant Content | Headers, footers, nav menus extracted by mistake |254| `4` | Indeterminate | Insufficient content to judge |255 256### Missing Content257Assessment of content completeness and extraction success:258 259| Component | Description | Path |260|-----------|-------------|------|261| Primary Code | Main missing content code | `eai_taxonomy.missing_content.primary.code` |262| Primary Label | Main missing content label | `eai_taxonomy.missing_content.primary.label` |263| Secondary Code | Alternative missing content code | `eai_taxonomy.missing_content.secondary.code` |264| Secondary Label | Alternative missing content label | `eai_taxonomy.missing_content.secondary.label` |265 266**Possible Values:**267| Code | Label | Description |268|------|-------|-------------|269| `-1` | Abstain | Unable to determine |270| `0` | No Missing Content | Complete and coherent text |271| `1` | Truncated Snippets | Obvious "...", incomplete paragraphs, cut-off text |272| `2` | Click Here References | "Download here", "Click here" without linked content |273| `3` | Incoherent Flow | Unreadable or illogical flow due to missing context |274| `4` | Missing Images or Figures | Placeholders or references to missing visual content |275| `5` | Missing Referenced Data | References to absent tables/datasets (e.g., "See Table 3") |276| `6` | Indeterminate | Insufficient content to judge |277 278### Text Structure Information279 280| Field | Type | Description | Path |281|-------|------|-------------|------|282| Line Start Indices | `List[Int32]` | Starting indices of each line | `line_start_n_end_idx.line_start_idx` |283| Line End Indices | `List[Int32]` | Ending indices of each line | `line_start_n_end_idx.line_end_idx` |284 285</details>286 287<details>288<summary><strong>Content Quality Dimensions</strong></summary>289 290Quality assessment inspired by NaturalReasoning and FineWeb efforts to categorize web data by information sophistication.291 292### Reasoning Depth293Assesses the complexity and sophistication of logical reasoning in the document:294 295| Component | Description | Path |296|-----------|-------------|------|297| Primary Code | Main reasoning depth code | `eai_taxonomy.reasoning_depth.primary.code` |298| Primary Label | Main reasoning depth label | `eai_taxonomy.reasoning_depth.primary.label` |299| Secondary Code | Alternative reasoning depth code | `eai_taxonomy.reasoning_depth.secondary.code` |300| Secondary Label | Alternative reasoning depth label | `eai_taxonomy.reasoning_depth.secondary.label` |301 302**Possible Values:**303| Code | Label | Description |304|------|-------|-------------|305| `-1` | Abstain | Unable to determine |306| `1` | No Reasoning | Facts present but no evidence of reasoning |307| `2` | Basic Reasoning | Basic analysis with minimal explanation and summarization |308| `3` | Intermediate Reasoning | Some logical steps connecting ideas and structured thinking |309| `4` | Advanced Reasoning | Multi-step reasoning and thorough analysis with well-developed explanations |310| `5` | Exceptional Reasoning | Novel abstractions, theoretical frameworks, long chain-of-thought, original insights, or proofs |311| `6` | Indeterminate | Insufficient context to judge |312 313### Technical Correctness314Evaluates the accuracy and precision of technical information:315 316| Component | Description | Path |317|-----------|-------------|------|318| Primary Code | Main technical correctness code | `eai_taxonomy.technical_correctness.primary.code` |319| Primary Label | Main technical correctness label | `eai_taxonomy.technical_correctness.primary.label` |320| Secondary Code | Alternative technical correctness code | `eai_taxonomy.technical_correctness.secondary.code` |321| Secondary Label | Alternative technical correctness label | `eai_taxonomy.technical_correctness.secondary.label` |322 323**Possible Values:**324| Code | Label | Description |325|------|-------|-------------|326| `-1` | Abstain | Unable to determine |327| `1` | Technically Flawed | Significant errors undermining content validity |328| `2` | Partially Correct | Some correctness but contains flaws, omissions, or errors |329| `3` | Mostly Correct | Technical correctness with minor flaws or incomplete explanations |330| `4` | Highly Correct | High technical correctness with precise definitions and clear explanations |331| `5` | Exceptionally Correct | Exceptional technical correctness with formal proofs and flawless content |332| `6` | Not Applicable/Indeterminate | No technical content or insufficient context |333 334### Education Level335Assesses the appropriate educational background required to comprehend the content:336 337| Component | Description | Path |338|-----------|-------------|------|339| Primary Code | Main education level code | `eai_taxonomy.education_level.primary.code` |340| Primary Label | Main education level label | `eai_taxonomy.education_level.primary.label` |341| Secondary Code | Alternative education level code | `eai_taxonomy.education_level.secondary.code` |342| Secondary Label | Alternative education level label | `eai_taxonomy.education_level.secondary.label` |343 344**Possible Values:**345| Code | Label | Description |346|------|-------|-------------|347| `-1` | Abstain | Unable to determine |348| `1` | General Audience | Accessible to anyone with basic literacy; simple terms |349| `2` | High School Level | Requires high school education; specialized terminology explained for non-experts |350| `3` | Undergraduate Level | Requires college education; uses specialized terminology and assumes background knowledge |351| `4` | Graduate/Expert Level | Requires graduate education or domain expertise; assumes deep background knowledge |352| `5` | Indeterminate | Insufficient content to judge educational level |353 354</details>355 356<details>357<summary><strong>Metadata</strong></summary>358 359## Metadata Structure360 361The `metadata` field contains a nested structure with web archive information:362 363| Field | Type | Description | Path |364|-------|------|-------------|------|365| **URL Information** | | | |366| URL | `String` | Original URL of the document | `metadata.url` |367| Source Domain | `String` | Domain name of the source | `metadata.source_domain` |368| Snapshot ID | `String` | Identifier for the web archive snapshot | `metadata.snapshot_id` |369| **WARC Metadata** | | WARC (Web ARChive) format metadata | |370| Content Length | `String` | Size of the content | `metadata.warc_metadata.Content-Length` |371| Content Type | `String` | MIME type of the content | `metadata.warc_metadata.Content-Type` |372| Block Digest | `String` | Checksum of the WARC block | `metadata.warc_metadata.WARC-Block-Digest` |373| Concurrent To | `String` | Related WARC records | `metadata.warc_metadata.WARC-Concurrent-To` |374| Date | `String` | Timestamp of the crawl | `metadata.warc_metadata.WARC-Date` |375| IP Address | `String` | Source server IP address | `metadata.warc_metadata.WARC-IP-Address` |376| Payload Type | `String` | Identified content type | `metadata.warc_metadata.WARC-Identified-Payload-Type` |377| Payload Digest | `String` | Checksum of the payload | `metadata.warc_metadata.WARC-Payload-Digest` |378| Record ID | `String` | Unique WARC record identifier | `metadata.warc_metadata.WARC-Record-ID` |379| Target URI | `String` | Original target URL | `metadata.warc_metadata.WARC-Target-URI` |380| Truncated | `String` | Truncation status | `metadata.warc_metadata.WARC-Truncated` |381| Type | `String` | WARC record type | `metadata.warc_metadata.WARC-Type` |382| Warcinfo ID | `String` | Associated warcinfo record | `metadata.warc_metadata.WARC-Warcinfo-ID` |383| **Additional Info** | | | |384| WARC Info | `String` | Additional WARC information | `metadata.warc_info` |385 386</details>387 388<details>389<summary><strong>Quality Signals</strong></summary>390 391The dataset includes two comprehensive quality assessment frameworks:392 393## Red Pajama v2 Quality Metrics394 395Text quality indicators derived from the Red Pajama v2 filtering pipeline:396 397### Content Structure Metrics398| Metric | Description | Path |399|--------|-------------|------|400| Original Length | Original document length | `quality_signals.red_pajama_v2.ccnet_original_length` |401| Original Lines | Number of lines in original document | `quality_signals.red_pajama_v2.ccnet_original_nlines` |402| Sentence Count | Total sentence count | `quality_signals.red_pajama_v2.rps_doc_num_sentences` |403| Word Count | Total word count | `quality_signals.red_pajama_v2.rps_doc_word_count` |404| Mean Word Length | Average word length | `quality_signals.red_pajama_v2.rps_doc_mean_word_length` |405 406### Language Quality Metrics407| Metric | Description | Path |408|--------|-------------|------|409| Stop Word Fraction | Proportion of stop words | `quality_signals.red_pajama_v2.rps_doc_stop_word_fraction` |410| Unique Words Fraction | Fraction of unique words | `quality_signals.red_pajama_v2.rps_doc_frac_unique_words` |411| All Caps Words | Fraction of words in all capitals | `quality_signals.red_pajama_v2.rps_doc_frac_all_caps_words` |412| Non-Alphabetic Words | Fraction of non-alphabetic words | `quality_signals.red_pajama_v2.rps_doc_frac_no_alph_words` |413| Unigram Entropy | Entropy measure of word distribution | `quality_signals.red_pajama_v2.rps_doc_unigram_entropy` |414 415### Content Pattern Analysis416| Metric | Description | Path |417|--------|-------------|------|418| Curly Bracket Density | Curly bracket density (code indicator) | `quality_signals.red_pajama_v2.rps_doc_curly_bracket` |419| Symbol-to-Word Ratio | Symbol-to-word ratio | `quality_signals.red_pajama_v2.rps_doc_symbol_to_word_ratio` |420| Ellipsis Line Endings | Lines ending with ellipsis | `quality_signals.red_pajama_v2.rps_doc_frac_lines_end_with_ellipsis` |421| Lorem Ipsum Detection | Lorem ipsum text detection | `quality_signals.red_pajama_v2.rps_doc_lorem_ipsum` |422| Offensive Content | Potentially offensive content detection | `quality_signals.red_pajama_v2.rps_doc_ldnoobw_words` |423| UT1 Blacklist | UT1 blacklist filtering score | `quality_signals.red_pajama_v2.rps_doc_ut1_blacklist` |424 425### Duplication Detection426| Metric | Description | Path |427|--------|-------------|------|428| 5-gram Duplication | Character-level duplication for 5-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_5grams` |429| 6-gram Duplication | Character-level duplication for 6-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_6grams` |430| 7-gram Duplication | Character-level duplication for 7-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_7grams` |431| 8-gram Duplication | Character-level duplication for 8-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_8grams` |432| 9-gram Duplication | Character-level duplication for 9-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_9grams` |433| 10-gram Duplication | Character-level duplication for 10-grams | `quality_signals.red_pajama_v2.rps_doc_frac_chars_dupe_10grams` |434| Top 2-gram Coverage | Most frequent 2-gram coverage | `quality_signals.red_pajama_v2.rps_doc_frac_chars_top_2gram` |435| Top 3-gram Coverage | Most frequent 3-gram coverage | `quality_signals.red_pajama_v2.rps_doc_frac_chars_top_3gram` |436| Top 4-gram Coverage | Most frequent 4-gram coverage | `quality_signals.red_pajama_v2.rps_doc_frac_chars_top_4gram` |437 438### Domain Importance Scores439| Metric | Description | Path |440|--------|-------------|------|441| Books Importance | Similarity to book content | `quality_signals.red_pajama_v2.rps_doc_books_importance` |442| Books Importance (Length Corrected) | Length-corrected books similarity | `quality_signals.red_pajama_v2.rps_doc_books_importance_length_correction` |443| OpenWebText Importance | Similarity to OpenWebText | `quality_signals.red_pajama_v2.rps_doc_openwebtext_importance` |444| OpenWebText Importance (Length Corrected) | Length-corrected OpenWebText similarity | `quality_signals.red_pajama_v2.rps_doc_openwebtext_importance_length_correction` |445| Wikipedia Importance | Similarity to Wikipedia | `quality_signals.red_pajama_v2.rps_doc_wikipedia_importance` |446| Wikipedia Importance (Length Corrected) | Length-corrected Wikipedia similarity | `quality_signals.red_pajama_v2.rps_doc_wikipedia_importance_length_correction` |447 448## FastText Classification Scores449 450Domain and content type classification probabilities:451 452| Metric | Description | Path |453|--------|-------------|------|454| DCLM Score | DataComp-LM classifier score | `quality_signals.fasttext.dclm` |455| English Confidence | English language confidence | `quality_signals.fasttext.english` |456| Educational Content | Educational content approximation | `quality_signals.fasttext.fineweb_edu_approx` |457| General Math | General mathematics content | `quality_signals.fasttext.eai_general_math` |458| Web Math | OWM Web-based mathematics content | `quality_signals.fasttext.eai_open_web_math` |459| Code Content | Code content detection | `quality_signals.fasttext.eai_web_code` |460 461</details>462 463## How to Load the Dataset464 465This section provides examples of how to load the `EssentialAI/essential-web-v1.0` dataset using different Python libraries and frameworks.466 467### Using Hugging Face Datasets (Standard Method)468 469The simplest way to load the dataset is using the Hugging Face `datasets` library:470 471```python472from datasets import load_dataset473 474# Load the entire dataset475dataset = load_dataset("EssentialAI/essential-web-v1.0")476 477# View dataset structure478print(dataset)479print(f"Number of examples: {len(dataset['train'])}")480```481 482You can also load the dataset in streaming mode to avoid downloading the entire dataset at once:483 484```python485from datasets import load_dataset486 487# Load in streaming mode488dataset = load_dataset("EssentialAI/essential-web-v1.0", streaming=True)489data_stream = dataset["train"]490 491# Iterate through examples492for example in data_stream.take(5):493    print(example)494```495 496### Using PySpark497 498For large-scale distributed processing, you can load the dataset using PySpark with the `pyspark_huggingface` library:499 500```python501# First install the required library:502# pip install pyspark_huggingface503 504import pyspark_huggingface505from pyspark.sql import SparkSession506 507# Initialize Spark session508spark = SparkSession.builder.appName("EAI-Taxonomy-Web").getOrCreate()509 510# Load the dataset using the "huggingface" data source511df = spark.read.format("huggingface").load("EssentialAI/essential-web-v1.0")512 513# Basic dataset exploration514print(f"Dataset shape: {df.count()} rows, {len(df.columns)} columns")515df.show(10)516df.printSchema()517 518# Load only specific columns for efficiency519df_subset = (520    spark.read.format("huggingface")521    .option("columns", '["column1", "column2"]')  # Replace with actual column names522    .load("EssentialAI/essential-web-v1.0")523)524 525# Run SQL queries on the dataset526df.createOrReplaceTempView("eai_web_dataset")527result = spark.sql("""528    SELECT COUNT(*) as total_examples529    FROM eai_web_dataset530""")531result.show()532```533 534### Using Daft535 536Daft provides a modern DataFrame library optimized for machine learning workloads. You can load the dataset directly from Hugging Face:537 538```python539import daft540 541# Load the entire dataset542df = daft.read_parquet("hf://datasets/EssentialAI/essential-web-v1.0")543 544# Basic exploration545print("Dataset schema:")546df.schema()547 548print("First 5 rows:")549df.show(5)550```551 552If you need to access private datasets or use authentication:553 554```python555import daft556from daft.io import IOConfig, HTTPConfig557 558io_config = IOConfig(http=HTTPConfig(bearer_token="your_token"))559df = daft.read_parquet("hf://datasets/EssentialAI/essential-web-v1.0", io_config=io_config)560```561 562### Installation Requirements563 564Make sure you have the required libraries installed:565 566```bash567# For Hugging Face datasets568pip install datasets569 570# For PySpark with Hugging Face integration571pip install pyspark_huggingface572 573# For Daft574pip install daft575```576 577## 📜 License578 579Essential-Web-v1.0 contributions are made available under the [ODC attribution license](https://opendatacommons.org/licenses/by/odc_by_1.0_public_text.txt); however, users should also abide by the [Common Crawl - Terms of Use](https://commoncrawl.org/terms-of-use). We do not alter the license of any of the underlying data.580 581## 📝 Citation582 583```bibtex584@misc{ai2025essentialwebv1024ttokens,585      title={Essential-Web v1.0: 24T tokens of organized web data}, 586      author={Essential AI and : and Andrew Hojel and Michael Pust and Tim Romanski and Yash Vanjani and Ritvik Kapila and Mohit Parmar and Adarsh Chaluvaraju and Alok Tripathy and Anil Thomas and Ashish Tanwer and Darsh J Shah and Ishaan Shah and Karl Stratos and Khoi Nguyen and Kurt Smith and Michael Callahan and Peter Rushton and Philip Monk and Platon Mazarakis and Saad Jamal and Saurabh Srivastava and Somanshu Singla and Ashish Vaswani},587      year={2025},588      eprint={2506.14111},589      archivePrefix={arXiv},590      primaryClass={cs.CL},591      url={https://arxiv.org/abs/2506.14111}, 592}593```