data-is-better-together/fineweb-c

FineWeb-C: Educational content in many languages, labelled by the community

<center> <img src="https://huggingface.co/spaces/data-is-better-together/fineweb-communications-pack/resolve/main/fineweb-c-card-header.png" alt="FineWeb 2: A sparkling update with 1000s of languages"> </center>

Multilingual data is better together!

Note: We are not actively working on this project anymore. You can continue to contribute annotations and we'll occasionally refresh the exported data.

What is this?

FineWeb-C is a collaborative, community-driven project that expands upon the FineWeb2 dataset. The goal is to create high-quality educational content annotations across hundreds of languages.

By enhancing web content with these annotations, we aim to improve the development of Large Language Models (LLMs) in all languages, making AI technology more accessible and effective globally.

The annotations in this dataset will help train AI systems to automatically identify high-quality educational content in more languages and in turn help build better Large Language Models for all languages.

What the community is doing:

• —For a given language, look at a page of web content from the FineWeb2 dataset in Argilla.
• —Rate how educational the content is.
• —Flag problematic content i.e. content that is malformed or in the wrong language.

Once a language reaches 1,000 annotations, the dataset will be included in this dataset! Alongside rating the educational quality of the content, different language communities are discussing other ways to improve the quality of data for their language in our Discord discussion channel.

What's been done so far?

So far 466 members of the Hugging Face community have submitted 58,188 annotations.

The following languages have reached the 10 annotation threshold to be included in the dataset.

You can help contribute to the dataset [here](https://huggingface.co/spaces/data-is-better-together/fineweb-c).

Note: We are not actively supporting this effort anymore but you can continue to contribute annotations and we'll occasionally refresh the exported data.

Below is an overview of the number of annotations submitted for each language (updated daily).

<iframe src="https://huggingface.co/datasets/data-is-better-together/fineweb-c-progress/embed/sql-console/dhn8hw-" frameborder="0" width="100%" height="560px"></iframe>

Why are we doing this?

There are many languages in the world where no high quality LLMs exist. Having high quality data is a central part of building high quality LLMs. FineWeb2 is a crucial step in improving the availability of high quality data for many languages. We plan to go a step further.

FineWeb-Edu for every language?

FineWeb-Edu is a dataset built on the original FineWeb dataset. The dataset was constructed by developing an educational quality classifier using annotations generated by LLama3-70B-Instruct and using this classifier to retain only the most educational web pages.

FineWeb-Edu outperforms FineWeb on popular benchmark. Crucially, using this approach reduces the amount of data needed to train a high quality LLM reducing the barrier to building a high quality LLM for many languages.

We want to make it possible to build FineWeb-Edu datasets for all the worlds languages. To do this we need annotations in order to train an educational quality classifier.

This in turn will allow us to build the next generation of Large Language Models for many languages.

Why not use LLMs to annotate the data?

For high resources languages, using an LLM to generate educational quality annotations can be a good solution. However, for many languages LLMs are not able to generate high quality annotations — or we don't have enough data to validate whether the annotations are correct.

How can I help?

You can help by contributing to the dataset here and join the community discussions in Discord!

Why would I bother to contribute to this dataset?

Your contributions directly shape the future of AI in your language. Here's why this matters:

1. 1.Break the AI language barrier: Most commercial AI companies focus on profitable languages, leaving many communities behind. Your work helps bring AI capabilities to more languages.

1. 1.Keep it open: Unlike proprietary datasets locked away by companies, FineWeb2-C is an open dataset. This means anyone can use it to build AI systems that truly serve their community's needs. Through this open approach we also learn about which approaches work best for different languages.

1. 1.Be part of something bigger: Just as Wikipedia showed how volunteers can build invaluable resources, the Hugging Face community has created numerous open models and datasets. You're joining a movement to democratize AI technology.

Every annotation counts. Whether you can contribute ten minutes or ten hours, your input helps build a more inclusive future for AI technology 🤗

Who contributed to this dataset so far?

These are the top 10 contributors to this release of the dataset. Make sure to give them a follow on the Hub to show your appreciation!

Data work is the under appreciated foundation of AI and ML. This dataset is built by the community for the community. Below is a leaderboard that is updated daily and shows all the contributors to this annotation effort.

<iframe src="https://huggingface.co/datasets/data-is-better-together/fineweb-c-progress/embed/sql-console/DJ2n1Z0" frameborder="0" width="100%" height="560px"></iframe>

Language-specific Contributors

Below you can find a list of all the contributors to this release of the dataset for each language ❤️

<details> <summary>Detailed Contributor Statistics for each language</summary>

Assamese (asm_Beng)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Assamese (asm_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Asturian (ast_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Bangla (ben_Beng)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Bashkir (bak_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Basque (eus_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Bavarian (bar_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Bhojpuri (bho_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Breton (bre_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Bulgarian (bul_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Cantonese (yue_Hani)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Catalan (cat_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Crimean Tatar (crh_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Cusco Quechua (quz_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Czech (ces_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Danish (dan_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Dutch (nld_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Egyptian Arabic (arz_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Faroese (fao_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Filipino (fil_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Finnish (fin_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

French (fra_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Galician (glg_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Ganda (lug_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

German (deu_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Goan Konkani (gom_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Gujarati (guj_Gujr)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Hindi (hin_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Hindi (hin_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Hungarian (hun_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Indonesian (ind_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Italian (ita_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Japanese (jpn_Jpan)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Kashmiri (kas_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Kinyarwanda (kin_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Korean (kor_Hang)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Latin (lat_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Lezghian (lez_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Ligurian (lij_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Lithuanian (lit_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Low German (nds_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Malayalam (mal_Mlym)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Mandarin Chinese (cmn_Hani)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Marathi (mar_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Middle High German (gmh_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Min Nan Chinese (nan_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Moroccan Arabic (ary_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Najdi Arabic (ars_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Nepali (individual language) (npi_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Nepali (individual language) (npi_Deva)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Nigerian Pidgin (pcm_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

North Levantine Arabic (apc_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Northern Uzbek (uzn_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Northern Uzbek (uzn_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Norwegian Bokmål (nob_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Old High German (goh_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Palatine German (pfl_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Pennsylvania German (pdc_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Persian (fas_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Polish (pol_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Portuguese (por_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Romanian (ron_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Russian (rus_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Scots (sco_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Serbian (srp_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Serbian (srp_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Sinhala (sin_Sinh)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Slovak (slk_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Somali (som_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Southern Pashto (pbt_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Spanish (spa_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Standard Arabic (arb_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Standard Arabic (arb_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Standard Estonian (ekk_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Standard Latvian (lvs_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Standard Malay (zsm_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Swedish (swe_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Swiss German (gsw_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Tamil (tam_Taml)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Tatar (tat_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Telugu (tel_Telu)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Thai (tha_Thai)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Tigrinya (tir_Ethi)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Toki Pona (tok_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Tunisian Arabic (aeb_Arab)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Turkish (tur_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Udmurt (udm_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Ukrainian (ukr_Cyrl)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Upper Sorbian (hsb_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Vietnamese (vie_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

West Flemish (vls_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

Yoruba (yor_Latn)

<details> <summary>User Statistics Table (Minimum 1 submissions)</summary>

</details>

</details>

Using this dataset

The dataset has a default config that contains all the language and configs per language.

To download the dataset using the Hugging Face datasets library, you can use the following code:

from datasets import load_dataset

dataset = load_dataset("data-is-better-together/fineweb-c-edu")

To download a specific language, you can use the following code:

dataset = load_dataset("data-is-better-together/fineweb-c-edu", name="cmn_Hani")

You can also download the dataset using Pandas

import pandas as pd

# Login using e.g. `huggingface-cli login` to access this dataset
df = pd.read_parquet("hf://datasets/data-is-better-together/fineweb-c-edu/arb_Arab/train-00000-of-00001.parquet")

or polars

import polars as pl

# Login using e.g. `huggingface-cli login` to access this dataset
df = pl.read_parquet('hf://datasets/davanstrien/fineweb-c-exported-data-test/arb_Arab/train-00000-of-00001.parquet')

Annotations with source metadata

The with_metadata config contains the same 45,160 annotation rows as default, with 58 columns including source URLs, language scores, duplicate-cluster sizes, complete selected-source records and versioned join provenance. All eight original annotation columns are preserved. Existing default and language configs keep their current data.

from datasets import load_dataset

dataset = load_dataset(
    "data-is-better-together/fineweb-c", name="with_metadata", split="train"
)

Source metadata is recovered for 43,086 rows (95.4%); 2,073 unmatched rows and one ambiguous row are retained with null selected-source fields. Most links refer to pinned historical sample datasets. Full-FineWeb2 membership was separately checked only for Kinyarwanda and Tatar. Labels continue to describe the original text, including when a later source version differs.

See metadata documentation for join rules, every added column, coverage by language, pinned sources and downloadable candidate audit tables.

Data Fields

The dataset contains the following columns:

The main things to note (we'll update this as we get more data)

• —Some languages already have multiple annotations per page. So far we haven't done any processing on these rows so people are free to calculate the agreement of the annotators in whatever way they want.
• —For languages with many active annotators, we may increase the overlap of annotations over time to further improve the quality of the dataset.
• —Some languages contain many problematic content labels. These often occur when the language detection was not correct. There is a problematic_content_label_present boolean column that indicates if the page contains at least one problematic content label. If you want to remove these rows you can do so by filtering on this column. Alternatively, you can use the problematic_content_label_agreement column to filter on the agreement of the annotators i.e. only remove rows where the annotators agree on the problematic content label. For many of the most active language efforts we're working with the community to improve the quality of the data so we hope the number of problematic content labels will decrease over time.

Licensing Information

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.

Citation

Citation information needs to be added

Last Updated

2025-07-08