Yahoo-Finance-News/FineWeb-2023

FineWeb-Edu 2023 — Cleaned and Shuffled

This dataset is a year-specific, cleaned, shuffled, and sharded release derived from HuggingFaceFW/fineweb-edu. It contains English educational web text collected in Common Crawl snapshots whose dump identifier belongs to 2023.

This is not a news-only dataset. The year refers to the Common Crawl capture year, not necessarily the page's publication year.

Dataset summary

The repository also contains manifest.json, which records the byte size, row count, and token total for every output shard.

Schema

Each Parquet file has exactly four columns in this order:

Cleaning and preparation

The release was produced from the upstream dataset at the fixed revision:

87f09149ef4734204d70ed1d046ddc9ca3f2b8f9

The processing pipeline performed the following steps:

1. 1.Selected the complete list of upstream Parquet shards assigned to 2023 and shuffled their processing order using a fixed seed.
2. 2.Streamed the columns url, dump, text, and token_count from each source shard.
3. 3.Rejected a row only when:
4. 4.text was missing, was not a string, or contained only whitespace;
5. 5.dump did not begin with CC-MAIN-2023-; or
6. 6.token_count was not a positive integer.
7. 7.Renamed url to source, set date to 2023, and retained text and token_count unchanged.
8. 8.Shuffled rows within processing batches with deterministic seeds.
9. 9.Assigned every cleaned row to a deterministic random bucket using a hash of source, date, text, token_count, and the annual seed 2065.
10. 10.Wrote the buckets as Zstandard-compressed Parquet files targeting approximately 1.5 GB per shard.

No additional deduplication, news classification, text normalization, language filtering, or quality threshold was added beyond the upstream FineWeb-Edu processing and the validation rules above. No token cap or sampling limit was applied.

Verification

Before publication, the pipeline:

• —checked the four-column Arrow schema of every output shard;
• —reopened every completed shard and verified its row count;
• —recomputed the sum of token_count for each shard;
• —verified that final row and token totals matched the cleaned intermediate data; and
• —published the final folder only after all shards passed validation.

The deterministic bucket assignment makes the partitioning reproducible for the same cleaned input, software behavior, and seed. Row order inside a bucket should not be treated as a stable identifier.

Loading the data

Because the dataset is large, streaming is recommended:

from datasets import load_dataset

dataset = load_dataset(
    "Yahoo-Finance-News/FineWeb-2023",
    split="train",
    streaming=True,
)

first_row = next(iter(dataset))
print(first_row.keys())
# dict_keys(['source', 'date', 'text', 'token_count'])

Source, license, and content notice

The text originates from web pages collected by Common Crawl and included in FineWeb-Edu. The source column provides the original URL recorded by FineWeb-Edu. This release follows the upstream odc-by dataset license. Underlying web content may remain subject to rights held by its original authors or publishers. Users are responsible for evaluating content, privacy, copyright, and suitability for their intended use.

For details about the upstream dataset's construction and educational-quality filtering, consult the FineWeb-Edu dataset card.