CoolFace
Datasetpublic

Michaelyya/fineweb-edu-v2-FastT

FineWeb-Edu v2 - FastText Climate Filtered A climate and environment-focused subset of sraj/finewebedu-climate-v2, further filtered using a trained FastText binary classifier. Overview This dataset applies a supervised FastText climate classifier to the FineWeb-Edu climate v2 dataset. Each record includes a climate probability score from the classifier, providing a confidence measure for climate relevance. Pipeline Source:… See the full description on the dataset page: https://huggingface.co/datasets/Michaelyya/fineweb-edu-v2-FastT.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
0likes277downloads
Dataset Card

FineWeb-Edu v2 - FastText Climate Filtered

A climate and environment-focused subset of sraj/finewebedu-climate-v2, further filtered using a trained FastText binary classifier.

Overview

This dataset applies a supervised FastText climate classifier to the FineWeb-Edu climate v2 dataset. Each record includes a climate probability score from the classifier, providing a confidence measure for climate relevance.

Pipeline

  1. 1.Source: sraj/finewebedu-climate-v2 (pre-filtered FineWeb-Edu)
  2. 2.Classifier: FastText supervised model trained on 10K GPT-labeled samples
  3. 3.Threshold: Records with climate_prob >= 0.5 are included

Fields

FieldTypeDescription
textstringThe document text
idstringOriginal document ID
urlstringSource URL
climate_probfloatFastText classifier probability (0-1)
sourcestringSource dataset identifier

Usage

python
from datasets import load_dataset

dataset = load_dataset("Michaelyya/fineweb-edu-v2-FastT")

Model Details

  • —Architecture: FastText supervised classifier
  • —Training: 10,000 samples labeled via GPT weak supervision
  • —Labels: Binary classification (climate vs. other)
  • —Hyperparameters: lr=0.5, epochs=25, wordNgrams=2, dim=100

License

MIT