JQL-AI/JQL-LLM-Edu-Annotations

1---2dataset_name: fineweb2-llm-annotated3pretty_name: JQL LLMs Multilingual Educational Quality Annotations4license: odc-by5source_license: Same as FineWeb2 (see upstream dataset)6size_categories:7- 10M<n<100M8language:9- bg10- cs11- hr12- mk13- pl14- sl15- sk16- sr17- uk18- da19- de20- is21- nl22- nn23- nb24- sv25- ca26- es27- fr28- ga29- gl30- it31- pt32- ro33- et34- fi35- hu36- lt37- lv38- el39- mt40- tr41- sq42- eu43- hy44- en45---46 47# 📚 JQL Educational Quality Annotations from LLMs48 49This dataset provides 17,186,606 documents with high-quality LLM annotations for evaluating the **educational value of web documents**, and serves as a benchmark for training and evaluating **multilingual LLM annotators** as described in the JQL [paper](https://arxiv.org/abs/2505.22232).50 51---52 53## 📝 Dataset Summary54 55  Multilingual document-level quality annotations scored on a 0–5 educational value scale by three state-of-the-art LLMs:56  Gemma-3-27B-it, Mistral-3.1-24B-it, and LLaMA-3.3-70B-it. Up to 500k documents per language from FineWeb2 are included.57  Annotations are aligned with human ratings and intended for quality estimation, distillation, and multilingual benchmark research.58 59## 🌐 Languages60 61  In total we included 35 European languages. Input documents are in their native language, but models were prompted and responded in English.62 63## 🧱 Dataset Structure: 64 65| Name             | Description                                         |66|------------------|-----------------------------------------------------|67| id               | Unique FW2 identifier for the document              |68| text             | Full textual content extracted from the webpage     |69| dum              | Common Crawl dump identifier from which the data originates              |70| url               | Source URL of the document              |71| date           | Timestamp indicating when the document was crawled (ISO 8601 format)              |72| file_path               | Path to the WARC file in the Common Crawl S3 bucket              |73| language       | ISO 639-3 language code of the document (e.g., deu)         |74| language_script               | Script used in the document (e.g., Latn for Latin script)              |75| language_score            | Confidence score of the language identification (float between 0 and 1)           |76| top_langs               | JSON string mapping detected language-script pairs to their scores              |77| minhash_cluster_size               | Number of documents in the deduplication cluster              |78| filter_reason               | Reason for filtering or deduplication (e.g., duplicated_5_n_grams), NaN if it would have been filtered              |79| edu_score           | Dictionary with per-model aggregated scores (modelname_score), **-1 if a invalid score was generated**   |80| aggregation | Dictionary with per-model aggregated type (modelname_type), either majority or average       |81 82## ✂️ Data Splits: 83 84  This dataset is not pre-split. Users can generate custom splits by:85  - Language86  - Model agreement87  - Prediction validity88  - Document length or other features 89 90## 🎯 Intended Use91 92  - Training multilingual document quality models93  - Benchmarking multilingual LLM performance94  - Distillation and teacher-student LLM training95  - Creating filters for noisy web-scale data96 97## ⚠️ Limitations: 98 99  - LLM-generated scores, not human-authored100  - Some predictions may be invalid or inconsistent101  - No domain control across documents102  - Educational value is a subjective, task-specific metric103 104## 📖 Citation105 106```bibtex107@article{ali2025judging,108    title     = {Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models},109    author    = {110      Mehdi Ali,111      Manuel Brack,112      Max Lübbering,113      Elias Wendt,114      Abbas Goher Khan,115      Richard Rutmann,116      Alex Jude,117      Maurice Kraus,118      Alexander Arno Weber,119      Felix Stollenwerk,120      David Kaczér,121      Florian Mai,122      Lucie Flek,123      Rafet Sifa,124      Nicolas Flores-Herr,125      Joachim Köhler,126      Patrick Schramowski,127      Michael Fromm,128      Kristian Kersting129    },130    year      = {2025},131    journal   = {arXiv preprint arXiv:2505:22232}132  }133```134 135 136## 🔗 Links: 137- Base Dataset: [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2)138- Related Work: [FineWeb2 LLM Judging Section](https://huggingface.co/papers/llm-quality-judging-fineweb2)